Improve heatmap implementation

[cpicanco]

Notes about the current blurred heat map implementation:

No blurr gives better feedback about the scaled coordinate system

I was wrong about that. I would rather say that to give control to the user, allowing blur gradation, is better to explore the data. Based on the histogram exploratory nature, i.e., there is no a priori or right interval/bin, you can conclude that there is no right amount of blurring as well.

"Ornamented" graphs usually are less informative than the "parsimonious" ones.

So, with the current implementation, indeed the blur does not work as an "ornament" at all. I don't know how to properly describe it, but it is essential to give proper feedback about the color mapping.

But of course, pupil community is not for "grumpy" scientists only. Pupil community is for everyone. Is that right?

But if you want, for some reason, with Pupil Player you could do anything regarding ornamented visualizations. That is what I mean.

[mkassner]

Well, to be honest. I don't think we have given this a lot of thought. I agree with your reasoning.

Should the blurring be adjustable? Does that even make sense for heat-map?

Maybe their is a correct amount of blurring (or distance) based on some paper?

[cpicanco]

Despite the fact that blurred heat maps are not difficult to find, you will find that some implementations makes it optional. I think this would be a good move, since with the current Pupil implementation, any amount of blur obfuscates the coordinate system scale. But it also gives smoother images. So, It should be optional, or there is a better answer?

[willpatera]

@cpicanco -- Good points. Would you be interested in testing out these ideas in code (time permitting of course)?

Notes/potential starting points:

• [ ] Make the number of histogram bins independent from X size and Y size -- in offline_reference_surface L245-L246. This could be exposed as a variable to the user per surface or globally.
• [ ] Toggle blur (on/off) switch for the user, or expose heatmap_detail as variable slider the user, or test alternative blurring functions and compare results (the current method is OpenCV's GaussianBlur) -- in offline_reference_surface L273
• [ ] Add user controls to the offline_marker_detector for the heat map variables. Options could be global (for all surfaces) or per surface. Additions would be made to offline_marker_detector in the neighborhood of L130-L147

Before adding all the UI controls, it might be useful just to quickly test blur parameters and alternative OpenCV filters (or one could make their own filter if you want that level of control :smile: ) Some links:

• OpenCV Documentation - Image Filtering
• OpenCV "Making your own linear filters" with filter2D
• OpenCV Python Tutorials - Smoothing Images (This site has a nice collection of examples and is quite well written)

[cpicanco]

@willpatera, I am very interested. Good opportunity to learn a little bit more about python, opencv and heatmaps. Congratulations for such comprehensive notes!

Please, can I ask you for a dead line? How much time you expect this task should be accomplished? Best.

[willpatera]

@cpicanco -- Great! There isn't any hard deadline (unless you'd benefit from one :grin: ). So, I think it would be a good (pressure free) area to evaluate different methods, test out ideas, and develop at your own pace. Looking forward to following and contributing to your process.

[cpicanco]

I first traceback the code, first to better understand the homograph implementation.

But for curiosity too, to see how easy it will be to return the p(x,y) pixels of each surface corner (top,left), (top, right), (bottom, left), (bottom right), including possibly negative ones. (Another history...)

Beginning with the basics. Now I know that "bins" means "intervals" (yep, language issue). As far as i know, not necessarily fixed ones considering the numpy implementation (histogram2d). (Can you think about an example that will demand variable intervals?)

In the current case, gaze_x, gaze_y intervals. As you can see, and is not very difficult to deduce, there is no precise rule to determine the size of each interval. The "volume" and "diversity" of the data will give the constraints. So one should be able to play around, to explore the data and then choose the "best fit" intervals.

About the homograph implementation, I am guessiing that the current .png is a rotated surface, say to a perpendicular angle. Is that right?

[cpicanco]

I have a question. Right now I am not sure if I should call the actual cv2.gaussianBlur as smoothing. It is clustering the coordinates as well. Is that right?

P.S.: Good reading

[cpicanco]

First approach

UI

The switch can be substituted for the zero in the slider.

Nearest interpolation

Histrogram bin 10, 10 size 240, 240

low len(xedge), len(yedge) gives good resolution because the image is being resized to fit the size of the surface

Blur/ Rouded interpolation

Note that bin 1, 1 and size, 240, 240 gives you something like the current implementation. no blur

code

def generate_heatmap(self,section):
        if self.cache is None:
            logger.warning('Surface cache is not build yet.')
            return

        # removing encapsulation
        x_bin, y_bin = self.heatmap_bins['x'], self.heatmap_bins['y']
        x_size, y_size = self.real_world_size['x'], self.real_world_size['y']

        # create equidistant edges based on the user defined interval/size
        x_bin = [x for x in xrange(0,int(x_size + 1), x_bin)]
        y_bin = [y for y in xrange(0,int(y_size + 1), y_bin)]

        all_gaze = []

        for c_e in self.cache[section]:
            if c_e:
                for gp in c_e['gaze_on_srf']:
                    all_gaze.append(gp['norm_gaze_on_srf'])

        if not all_gaze:
            logger.warning("No gaze data on surface for heatmap found.")
            all_gaze.append((-1., -1.))

        all_gaze = np.array(all_gaze)
        all_gaze *= [x_size, y_size]
        print all_gaze
        hist, xedge, yedge = np.histogram2d(all_gaze[:, 0], all_gaze[:, 1],
                                            bins = (x_bin, y_bin),
                                            # range = [[0, x_size], [0, y_size]],
                                            normed = False,
                                            weights = None)

        # numpy.histogram2d does not follow the Cartesian convention
        hist = np.rot90(hist)

        # smoothing/ rounded interpolation?
        if self.filter_blur:
            filter_size = (int(self.heatmap_detail * len(x_bin)/2)*2 +1)
            std_dev = filter_size /6.

            hist = cv2.GaussianBlur(hist, (filter_size, filter_size), std_dev)

        # scale convertion necessary for the colormapping
        maxval = np.amax(hist)
        if maxval:
            scale = 255./maxval
        else:
            scale = 0

        # colormapping
        hist = np.uint8(hist * (scale))
        c_map = cv2.applyColorMap(hist, cv2.COLORMAP_JET)

        # we need a 4 channel image to apply transparency
        x, y, channels = c_map.shape
        self.heatmap = np.ones((x, y, 4), dtype = np.uint8)

        # lets assign the color channels
        self.heatmap[:, :, :3] = c_map

        # alpha blend/transparency
        self.heatmap[:, :, 3] = 125

        self.filter_resize = True
        # here we approximate the image size trying to inventing as less data as possible
        # so resizing with a nearest-neighbor interpolation gives good results

        if self.filter_resize:
            inter = cv2.INTER_NEAREST
            dsize = (int(x_size), int(y_size)) 
            self.heatmap = cv2.resize(src=self.heatmap, dsize=dsize, fx=0, fy=0, interpolation=inter)

        # texturing
        self.heatmap_texture = create_named_texture(self.heatmap.shape)
        update_named_texture(self.heatmap_texture, self.heatmap)

[cpicanco]

I was wrong about that "No blurr gives better feedback about the scaled coordinate system."

Better say that control the blur gives some freedom to explore the data.

[willpatera]

Hi @cpicanco -- exciting update! Are you working on this in a branch within your fork of Pupil? If so, please share a link to the commit (if not please make a commit and share it :) so that I (and others) can try it out and make comments directly on the commits/code lines.

[cpicanco]

I just merged pupil-labs/pupil upstream to the master branch. https://github.com/cpicanco/pupil, https://github.com/cpicanco/pupil/commits/master,

"first approach commits"

[cpicanco]

Second approach,

This can help the user to estimate the size, will commit soon.

    def get_surface_vertices(self):
        """
            Returns 4 denormalized vertices (vx,vy) and its corresponding indexes (i).

            3 . . 2
            .     .
            .     .
            0 . . 1
        """
        if self.detected:
            denormalized_vertices = []
            surf_verts = ((0.,0.),(1.,0.),(1.,1.),(0.,1.))
            for (vx,vy),i in zip(self.ref_surface_to_img(np.array(surf_verts)),range(4)):
                vx,vy = denormalize((vx,vy),(self.img_shape[1],self.img_shape[0]),flip_y=True)
                denormalized_vertices.append([(vx,vy),i])
        return denormalized_vertices

    def estimate_size(self):
        if self.detected:
            for (vx, vy), i in self.get_surface_vertices(): 
                if i == 0: # bottomleft
                    x0, y0 = vx, vy
                if i == 1: # bottomright
                    x1, y1 = vx, vy
                if i == 3: # topleft
                    x3, y3 = vx, vy

            self.x_dist = math.hypot(x1 - x0, y1 - y0) 
            self.y_dist = math.hypot(x3 - x0, y3 - y0) 

            self.estimate_x_size()
            self.estimate_y_size()

    def estimate_x_size(self):
        print self.x_dist 

    def estimate_y_size(self):
        print self.y_dist 

and the ui of the offline_marker_detector

    s_menu.append(ui.Button('Estimate Size', s.estimate_size))
    s_menu.append(ui.Text_Input('x',s.real_world_size,label='X size', getter=s.estimate_x_size))
    s_menu.append(ui.Text_Input('y',s.real_world_size,label='Y size', getter=s.estimate_y_size))

[cpicanco]

@willpatera, the commits.

Estimate Size UI 6e05e6637d0976487e96385aaf9a511b07abc4e9 Estimate Size (function) fc5c78cd810dd268e266d54ad2a502c16ae387e0 First approach on Heatmaps UI 002ab5101a420580751e35f7749f0254284def0d First approach on Heatmaps (function) df755619e3eb8e7e139dedc9a1eec218fc5707d7

I think that it is a good opportunity to learn more about data analysis.

[willpatera]

Great!! I'll test it out later today.

[willpatera]

Hi @cpicanco -- Apologies for the delayed replies. I ran your code from your master branch and encountered a few issues that could lead to a crash, that can quickly fixed:

• variables filter_blur and filter_blur_detail in offline_marker_detector L137-138 need to be declared in __init__
• srt() is not a function in offline_reference_surface L273 and L276 I believe that you were trying to use square root sqrt() -- math is imported on L10 - so you could call math.sqrt() or when you import you could just import the two functions that you use from math -- like so from math import sqrt, hypot
• the list comprehension to create equidistant edges in offline_reference_surface on L288-289 will lead to x_bin = [0] if the step size is >= to the size -- which will lead to a crash.
• In the UI the X bin or Y bin parameter could be a misleading label, as the actual number of bins are calculated based on the size of the surface. So really this parameter is the step size.
• Bug - The size of the surfaces never gets updated and therefore the heatmap does not get updated, or can lead to a crash if the user changes the bin size to be > 1. I'd have to look into this further -- please see if you can replicate this.
• The user should be able to manually specify the surface size -- currently you can only use your estimation button when the surface is detected. This is because you have assigned a getter function to the Text_Input ui element.

General notes for your consideration:

• Its nice to have fine grain control over parameters. But we also need to make sure that we don't overload a user who doesn't care about some of these parameters. So it would be good to find a balance such that each surface in the sub-menu by default can have some of the advanced parameters hidden. Maybe an advanced menu per surface would be one solution here (however I would be hesitant to create further nesting).
• I really like the Blur boolean and the Blur gradation variable parameter slider.

[cpicanco]

Take your time @willpatera , take your time. So, did you like the estimate size idea?

• I am really concerned about something. For me seems to be possible distinguish the surface size (homography calculations) from the the heatmap size (for on screen presentation, eye inspection purposes). What do you think... @willpatera ?? Am I insane @mkassner ?
• I completely agree that the interface is overloaded, I really was not thinking about the usability. Instead, I was following your advice and just added some controls to test the new functionality. I will start to think about that. :)
• str() was used to return a string. The ui.Text_input "getter" function requires a string, or not? I have supposed that the conversion was necessary.
• I could not predict that with a "getter" function the text_input control would loose its input. How may I fix that? It is not a PyGlui issue, is it??
• That is right, currently it is a constant step size. Or interval size. Or bin size. Not number of steps, number of intervals, not number of bins. I will trust in your English. :)
• Honestly, I can't see a gooooood reason to maintain the "bins" separated. What do you think?? The Blur gradation seems to be good enough for this specific kind of eye inspection. I could not formulate some criteria to compare the two and reach a conclusion about which one is inventing less data than the other (in this case resizing with nearest interpolation). I am really interested in this question right now.
  • One would better benefit from practical export controls, for example, to resize the image to fit some specific resolution. This export controls could be presented only after a click on the export button, in a new window.
• Custom kernels (for the people who really want that level of control) could be better achieved with running from source, IMO, taking advantage of the open source project, Python simplicity and good documentation (when it is present). I will leave that for the computer science Pros. :)

[willpatera]

Hi @cpicanco

• I need to take another look at the estimation function -- however I still think that the user should be presented with a way to enter the real world scale as this is important as @mkassner mentions in #96
• UI - You're right, it's important to get the controls in there and test functionality first, then refine the UI after everything has been tested.
• Text_Input tries to conserve the input type. See pyglui ui_elements.pxi L615-625. So, the conversion was not necessary (however you did have a typo -- your code has srt not str :)
• Using getter ignores user input by design - the value could be updated by a variable.
• Another field for (bins/steps) - It might be best to leave this out as I agree gradation and the 'blur' toggle seems to give the insight and control over the visualization. But, I don't have a strong reason/criteria to support my opinion, other than from a UI standpoint.
• I agree with you on custom kernels -- this is something that can be done via source.
• Export controls idea - I'm not sure I understand entirely. You would like to be able to control the size of the .png generated?

[cpicanco]

@willpatera, sorry but I am working in another project. I will return to the present problem soon. I will be finishing the upgrade of my plugins to the v0.4x and solving some bugs in my stimulus control program.

---

• Export controls idea - I'm not sure I understand entirely. You would like to be able to control the size of the .png generated?

I was thinking about some journals that requires high resolution images. Yes, the resolution (size), and if possible an option to overlay the current frame (surface roi) to the heatmap image (I was reading the source and noticed that you guys already implemented something like this, for some reason it is commented).

[cpicanco]

I am not sure if something here should be merged. Please, make me know, I could make a pull request including the @willpatera recommendations. For me it is not urgent at all.

[willpatera]

@cpicanco -- The blur toggle and the blur gradation slider are very useful and self explanatory for users. If you make a pull request for those changes (minus additions for real world scale estimations) @mkassner and I will happily review it :smiley:

[cpicanco]

I found a study describing a "Standard heatmap algorithm" as follows:

Standard heatmap algorithm

The Gaussian topological function generates smoother heatmaps and thus, with no surprise, became popular among commercial products, like Ogama and SMI BeGaze. The Gaussian function is defined as below:

For each raw gaze data or fixation (x i , y i ) on a stimulus, the above function calculates intensities of surrounding pixels (x, y) of the stimulus image. The spread of pixels receiving more intensity from (x i , y i ) is defined as kernel size, the sum of all intensities on each pixel than generates an array of intensities. Color of the stimulus image pixel is based on a proportion between the pixel value and the maximum intensity.

Time complexity and quality of heatmap results depend on sigma and kernel size constant values. Sigma is known as variance. As illustrated in Figure 2, a low value of sigma creates a lot of small spots with wide cool area, while a high value results in very few hot spots and heatmap is colored uniformly. Kernel size is the second constant in the Gaussian function. Figure 3 shows that a narrow range creates many small heated spots, while a wide range gives a smoother heatmap. The rendering time will be longer if the kernel size is large because intensity of more pixels around gaze samples need to be computed. In this work, we test the kernel size in range of [100, 200] and the sigma in range of [25, 50].

http://dl.acm.org/citation.cfm?id=2578187

They are on RG.

[willpatera]

Hi @cpicanco -- thanks for the link, I will read through it :smile: