Generate SSA files automatically after recording

[garyfeng]

I need to reuse the old script to generate SSA files from the HDF5 files. This should be done right after the session, preferably automatically.

• sync the video and eye tracking data
• call the ssa generation script to read data from HDF5 and generate SSA
• save the ssa on the same directory as the video
• optionally provide a script to do all these and launch VLC to playback. If we run eye-tracking during playback, we have a post-task retrospective think-aloud session.

[garyfeng]

copied and tested 2 old python scripts:

C:\Confero\ConferoTrack\lib\data_processing>python hdf5_to_eye_log_commandline.p y -s Zaption C:\Confero\ConferoTrack\Results\Default_Experiment\iohub_events.hdf5

C:\Confero\ConferoTrack\lib\data_processing>python create_subtitle.py Zaption_ey e.log >Zaption_eye.ssa

The only thing missing is the time offset between the video and gaze data, which we can use the example in

vidEvtLookUp.py

I will write a script to put all these together.

[garyfeng]

will email Sol the SSA script, so that he can integrate the function in the appropriate post-processing script.

[isolver]

Does VLC use some time offset to the times given in the SSA file when presenting the SSA graphics? It seems like I have to add 1 second to the corrected times to get them to match ( testing by using mouse pos for SSA graph pos instead of eye).

[isolver]

The only issue I see is the offset that I mentioned. It must be coming from the VLC end of things. I have added a parameter to the script that can be used to apply an offset . It is set to 1.0 right now, which is what the offset added by VLC seems to be regardless of the video file I try (I have only tried 6 or 7 ).

[garyfeng]

Not that I am aware of. We may have added a some delay in our processing script, because in the old software setup we waited 1 sec between video recording and eye-tracking, I think. So if you are using the old script, there may be a constant or parameter somewhere.

On the other hand, the VLC gaze overlay doesn't need to be very accurate. I am happy if it's within a second or less.

[isolver]

I'm not using the old script other than the part that had the file format in it. Anyhow, with the offset fixup it looks very accuracy to me. ;)

So the script to create a SSA file is in the ConferoTrack/lib/data_processing folder.

It is called videvt2ssa.py . You have to be in that script folder when running it, so I added a .bat file at the root Confero folder called Confero Data Processing.bat.

If you run that, it opens a cmd prompt in the data processing folder. You can then type:

python.exe videvt2ssa.py [experiment_folder_name] [session_folder_name]

and it will convert each video in the session folder and make a .ssa file with the same name in that session folder.

There are 2 variables in the script itself that you may want to adjust when testing:

//The offset (in seconds) to apply to all event times so that they align with VLC playback using the SSA files created. VLC_OFFSET_CORRECTION=1.0

// If True, use mouse position, not gaze position, when creating SSA file. PLOT_MOUSE = True

[garyfeng]

sounds great.

[garyfeng]

Quick question - how is missing gaze data dealt in SSA? It seems from the playback that the eye would jump back to the screen center every so often. I suspect these are missing data. This can be confusing to the viewer.

There are two options we've used to deal with missing data in SSA:

1). Don't output anything for the frame. In this case the marquee will disappear. If the eye goes in and out a lot, this will create a lot of flickering.

2). Keep the last known gaze location for X seconds, where often X =1 second or so.

This helps to deal with brief missing data, blinks, etc.

Would you please look into this? Thanks!

[isolver]

Implemented option 1), makes most sense IMO.