Eye-tracking: Animate eye tracking data

[bastianilso]

This issue tracks the progress in animating the eye tracking data with Plotly.

[bastianilso]

Hi Thomas, thanks for the work so far, looks good.

Regarding the visualization, what does the big blue circle represent? (I assume the small blue points represent the current eye tracking samples?)

Right now i'm trying to show the active mole, i have some struggle since last week: I have a data frame with all the information But right now i still don't know how to plot it, i'm trying some stuff. You can check the code if you want on the git hub : https://github.com/ThomasDupin/WhackAMoleDataJune21/blob/main/whackamole_analysis.R#L621

As a starting point, I suppose you can plot it with the add_trace command - i suppose you already have this working? Then the next step is what we should pass on to frame. I suppose you are just using the time column there right?

However, most likely plotly will try to interpolate between frames. For making the proper visualization correspond to what happens in Whack-A-Mole VR, I think you will need to animate opacity. This way it looks like the mole appears, is visible, and then disappears, just like in the game. Alternatively, you could animate size (size 0 i guess is invisible), see e.g. the example in https://plotly.com/r/animations/

[ThomasDupin]

Hello Bastian,

The big blue circle is the mean X and Y of the eye tracking sample. The mean for one seconds of data. and the small point represent 10 sample per seconds.

Yeah i've tried to use an add trace exactly like you said. i will have a look at what you said ! Thank you

[bastianilso]

Hi thomas,

The big blue circle is the mean X and Y of the eye tracking sample. The mean for one seconds of data. and the small point represent 10 sample per seconds.

If it is the mean, how come the blue circle isnt "on top of" the smaller points then? At least thats what I would expect it to be.

[ThomasDupin]

Hello bastian ,

Here is the gif of the current state of the plot.

Firstly i've merged the dataframes, and i change the way i get the gaze data Now i get only one out of three line and it's limited to 10 sample per seconds.

As you can see with the gif the plot is doing the weird stuff because the mean gaze in blue seems to merge with the active mole in red and at about 30 sec i click on the legend and the mean gaze disappears so i don't know if you have some clue about this.

[bastianilso]

Thanks for the update, Thomas. What is your strategy for debugging this issue?

[ThomasDupin]

i've tried to run the trace separately but nothing really change, and right now i'm checking how the graph is doing if i remove the mean gaze but nothing much right now

[bastianilso]

@ThomasDupin some suggestions for debugging:

• Try to shorten your data to a tiny segment, so it is easier to compare what happens in the plot animation, with the data you pass to Plotly.
• Does applying a %>% filter change the behavior? or does it lead to an error?
• Are you currently using NA values? Instead of NA values, you could try to duplicate the values with e.g. %>% fill() but set the opacity to 0 (so it becomes invisible).
• Does applying a group make a difference? (see e.g. how to use this in this answer)

How come we can have a "working behavior" with the Patient Gaze, but not with the Mole Spawn Points / Mean Gaze? Does it have something to do with how the data structure was set up?

How many bugs are there in total here? Rather than try to fix them all at once, try to separate the issues from each other when we debug them.

Feel free to post new GIFs, they are helpful for understanding your process.

[ThomasDupin]

Ok thank you Bastian, i'm on it

[ThomasDupin]

Hello Bastian,

Here is an update about the plot.

I have changed a lot the data structure and as you can see below. I have replaced the NA with 0 but I have also attributed the opacity dynamically so if there is no mole the opacity will be 0 (I don't think this is working properly right now)

Now my main goal is to do a better merge because as you can see some line seems to be duplicating, in red this is the Active mole for the time 4.

Above its the data frame that I use for the plot and below this is the data frame where a retrieve the information about the mole. and as you can see there is some duplicate, because on the time 4 i have only one mole.

And also one last thing, for the meeting everyday at 8 am i don't have a teams link, because as i send you last week i'm stuck in france until the 14/15 January for my third injection it's a nightmare.

Have a nice day.

[bastianilso]

Thanks for the update @ThomasDupin . What functions are you using when merging datasets currently? maybe I can come with some suggestions, if I can see the datasets and your current choice for merging.

[ThomasDupin]

Okay i have edited my post above because i didn't realise that my gif was a short loop.

There is the two dataframe : The one with the data about the active mole

The one with the gaze stata :

I think i could try to get everything on the same dataframe instead of merging, because the lenght of the two dataframe is different so maybe something to check here

[bastianilso]

@ThomasDupin feedback on the animation: I think we would want to have the opacity set to 0 in a pattern like this:

XMole   YMole   time   opacity
0.000   0.000   0      0
3.370   0.6139   1      0
3.370   0.6139   1      1
3.370   0.6139   2      1
3.370   0.6139   3      1
3.370   0.6139   3      0
3.370   3.0861   3      0
3.370   3.0861   3      1
3.370   3.0861   4      1
3.370   3.0861   5      1
3.370   3.0861   6      1
3.370   3.0861   6      0
...

e.g. so that you always start on a 0, and end on a 0, for the given position.

As for merging, here is what I would do in your case:

• Both columns have the column time in common. I agree with your assertion that binding the rows is better, than merging. Especially if frame does not care whether you have several entries of the same time value - (do you know?).
• in this case try to use dplyr's bind_rows() to bind the two datasets together. and replace NAs with 0 afterwards.

[ThomasDupin]

Okay thank you for the feedback, i will start with the bind_rows() and once its done i will check with the opacity

[ThomasDupin]

Hello again Bastian !

Here is another update :

As you can see still not perfect, if you have some feedbacks i will be happy yo have it !

I don't know if it's from the data, but it seems that the plot become more accurate at about 15 seconds and after it seems to follow the mole quite nice. ( There is still some bizarre thing at time 30 )

I'm using this to combine my dataframe,

tested <- bind_rows(MoleActivated, GazeStatArranged)
tested <- arrange(tested, time)
tested$opacity[is.na(tested$opacity)] <- 0

This is the 3 lines that i'm using, the last one is for the opacity. I'm already attributing the opactiy before regarding if there is a mole or not, and this line replace the NA with 0 from the bind_rows().

As you can see above there is a mole at 0:0 who stay everytime, i'm sure the way i'm attributing the opacity is wrong, below its the line i'm using : add_trace(name="Active Mole", data=tested, x=~XMole, y=~YMole, type='scatter',frame=~time, marker=list(size=32, color = 'rgb(255, 0 , 0)'),opacity = tested$opacity) %>%

I'm sure it's from this because for the line with 0.0000 on X & Y the opacity is 0 so i guess there is something here. This is my priority right now i would like to fixe the opacity

I have another thing, the plot is too fast, because the duration is exactly 121 secondes, but the plot will run in about 60/70 sec so i will try to slow it down in order to be more accurate of a real game session.

[bastianilso]

@ThomasDupin again, I think the easiest way/most efficient way to really debug this, is to turn this into separate problems. To me, it looks like what you have is something that "sometimes" works, and "sometimes doesnt". It's difficulty to understand, when its all mixed up like that. Create some "reproducible" cases that are as short as possible, for each problem you are trying to solve. Are the problems only there when you combine everything? Or are they also present if you isolate the traces in their own plots?

Debugging Opacity: Fx how about cutting out a 5 second segment of your dataset, and focus on just making one active mole appear and dissapear, then for the next active mole to appear and then dissapear?

[ThomasDupin]

Hello @bastianilso,

Here is an update:

I was kind of blocked this morning and I couldn't debug my plot, I was feeling that something was wrong.

So, i have changed a lot of things, I wanted to try what we discuss this week about a different use of time. I'm trying to work with another format of time I’m using 1.0 1.1 1.2 ... instead of 1 2 3 I feel that this is way more accurate.

Bellow here is a new plot by using this method, I had to change a lot of things:

Firstly, I think that this is too slow, I have 10 sample per seconds so I think it might be a good idea to try to reduce to 5 per seconds.

I feel that the gaze is now more clear and more accurate than before, its looks way closer of the reality than the previous one but I’m not sure if it's correct.

Here is what my new data frame looks like. I underlined in red the line where a mole is active. And by using this method and the new format of time the opacity works way better, the opacity now changes in real time. I could now try to add an ID to each mole like we said before in order to tell plotly that they are different.

I will first reduce the length of my data frame in order to speed up the plot.

After when I think that I have the right data frame I will look at the active mole.

Best regards,

Thomas Dupin

[ThomasDupin]

Okay quick update,

I have managed to reduce the data to about 5 samples per seconds. It's already way better and faster, I can even reduce it to 3 samples per seconds if needed. This afternoon I will look into the active mole because, as you can check on the plot below, I feel like sometimes it’s really inaccurate I need to make plotly understand that each mole is different. So that’s my plan for this afternoon

[ThomasDupin]

Hello Bastian,

Quick update about the plot, i have a added id to the mole and changing the color depending of the id.

Here is what he look like :

i will continu to work on it because it seems that sometime the data is before the mole like the eye go to a location and go back to the center and after that the mole appear at the location where the eyes where before. So i will try to investigate that

[ThomasDupin]

To continu what i was saying before, especially at this time ( around 7 to 9 sec) the the gaze seems to predict the position for one Mole, i don't if it's just a coincidence.

[bastianilso]

Hi thomas, I don't understand what is going on with the green moles in this graph. why are they moving? they should have static positions, see gif below:

[ThomasDupin]

Yes i agree with you, i still think that plotly don't make the difference between each moles, i'm looking at it right now

[ThomasDupin]

Hello again Bastian,

I'm running out of solution for the mole, i tried to add unique ID to each mole like this:

So i change the opacity and the color dynamically for each mole, but plotly still not interpretate mole as individual. I have tried with an id different for each line who is equal of the row number, and another id who is the paste of X and Y of the mole both didn't work. I have a tested like you said with factor or without factor. But the result is approximatly the same.

I don't really know what to do. Maybe i should re start everything, i still think that the method that i'm using right now allow me to have a way better gaze.

I am working with a datafram with only the data of the mole to test some stuff this a bit better than before but as you can see most of the time a mole in 0.0 appear and i don't know why.

So if you have any tips or maybe i think it's a good idea to schedule a call or physical meeting in order to discuss everything and with that you can look throught it.

[ThomasDupin]

Hello again Bastian,

I have decided to work on shorter data frame, so each line have all the mole (2) and i set the opacity to one when i want to show a mole. I have tried to plot with the plot example code online with the data about continent

Here is the first gif so as you can see it's not really relevant.

Below you can find the code for the gif,

So, there is a mole who appear and disappear but, as you can see the mole appear at 0.3 but inside the data frame at 0.3 there is no mole everything is at 0 opacity.

And so, there is here the main plot with the changed data frame, and instead of changing one line he will change all the line to 1 opacity. so, for example at 0.4 I have one mole but he will display the 3 line because at 0.4 the opacity is changed to 1. This is the same problem that I had with the old data frame and that's a reason why now i work with time 1.0 1.1. Because i want to have individual line for each time.

I will keep trying to understand more how it work.

[bastianilso]

@ThomasDupin Let's do a short meeting on this tomorrow. I don't really understand why your plot would be empty for the first graph.

[ThomasDupin]

Okay perfect good idea, see you tomorrow

[bastianilso]

@ThomasDupin for the meeting tomorrow, I would like you to think of what the next logical steps to test would be for your plot and why. Then I will give feedback on your process in the short meeting tomorrow.

[ThomasDupin]

Okay, so the next steps to make sure that plot is working ? Or the next step to make it work or debug it ?

[ThomasDupin]

Looks pretty nice.

[bastianilso]

great @ThomasDupin, i agree, lets roll with this!

[ThomasDupin]

Here is the update,

I have managed to combine the gaze stat and the active mole, it was harder than expected and i find some buggy things, for example A dataframe like that will not work but if just change this order it will work so i need to be carefull with the order.

If you see multiple point on the plot at some times it's normal in order to make the plot work i had to fill a lot of data and manipulate a lot of NA values.

[ThomasDupin]

Hello Bastian here is an update, On the first gif this is my big dataframe with all the data, i'm trying to reduce it because we have around 41 different mole so at each time we have 41 entry this is too much. And as you can see this is impacting a lot the plot because everything is buggy.

The second plot is the exact same dataframe but i have sliced everything expect the data around 2.0 to 3.5 to see, and as you can see with less data everything is working fine

I think that the plot is overloaded ( when i have everything i have about 51 314 rows) now that i have everything i need and i know the plot can work i will try to reduce it without impacting to much the plot

[bastianilso]

@ThomasDupin maybe reduce the time to every half second or every second, instead of every 0.1 second, if you are struggling with too much data.

[ThomasDupin]

Hey @bastianilso, quick update:

Here is the result that I have right now, I have managed to get this amount of data. I have about 1 sample each second but with the 40 mole each time I have for 9 sec of data 700 rows. the plot doesn’t work with more. And as you can see at approximately 8.5 sec the plot loose in quality, but it seems to come from the data.

This is the data from D and you can see that multiple time I have 0.000 0.000 for moles.

Is it possible to schedule a meeting today or tomorrow because i'm strugging a bit to fix the plot.

[ThomasDupin]

Hey Bastian, here is the update:

Firstly, I have tried to document the code and I made a clear separation between the test code and the real plot code. Also, I added a lot of comment to explain each part of the code.

After that i work with a relatively small data frame to test with 3 moles:

As you can see it works fine but: This is the exact same data frame, but I have switched the mole who appears a 2.5 I placed her at 1. Everything is exactly the same, it has the same order etc... and it's not working and of course if I switch back the mole, it will work again so this is something that I don't really understand here. Because just by changing when the mole will appear the plot can stop working.

After that I looked into the mole who had 0.000 0.000 for coordinate. I found that they correspond to the event " Mole missed" so I have changed the way I get the mole data so now I only have the mole with the event " mole spawned ".

I figured out that the way I get the session data was wrong, I was missing a very small portion of data it was not really impacting but I still fixed it.

So, this the plot with all the changes to the mole, the time 9.2 is less buggy but he has the same wrong behaviour than before, and now I don't have any mole at 0.000 0.000.

[bastianilso]

This is the exact same data frame, but I have switched the mole who appears a 2.5 I placed her at 1. Everything is exactly the same, it has the same order etc... and it's not working and of course if I switch back the mole, it will work again so this is something that I don't really understand here. Because just by changing when the mole will appear the plot can stop working.

Could you please take screenshots of the dataframes side by side and highlight what you changed, which caused the issue to appear? Will make this easier to debug.

[ThomasDupin]

Hello again,

This frustrating, this morning I couldn't reproduce the case of yesterday, I didn’t touch the code but when I run it again like yesterday impossible to reproduce it, so I just moved on the next thing.

After a long time of analysis of why this is not working, I figured out that the data frame was not clean and I still had some mole with 0.000 0.000 positions so I removed them and i tried to get the cleanest data frame as possible, but as you can see it's still not working for one mole:

Below it's the data frame associated: Underlined in red this is the mole who is not working,

I have tried a lot of different thing and even if I change her position (The time when she appear) this specific mole fail every time:

Underlined in red this is the mole who is not working,

I'm made some test with the big data frame nothing really new: I just tried to run it with longer data or just see how it works overtime:

I will continue on this this afternoon, but I would also like to work on the documentation and my report.

[bastianilso]

Hi @ThomasDupin, how are things? let me know. :-) we should set some time off to go through your documented code next week.

[ThomasDupin]

Hello Bastian,

I didn't manage to fix the bug on the plot, so instead of staying blocked I’m working on my documentation and my internship report since yesterday because I really want to finish at least the documentation for today before leaving the internship.

And no problem for next week I am free on Tuesday if you want.