log_sensor_table missing entries in log_file_path

[lwhite1]

Description of problem from @sid4py via slack: after yesterday's session I noticed that we seem to be missing the calibration edf file generated by eyetracker during calibration task. Edit: After a little more looking we are actually getting a file called 'test' in the log_sensor_file table but this file does not actually exist in the data from the session. The Calibration edf file also exists in log_sensor_file table but is similarly is missing from session data. (edited)

Siddharth Patel 9:43 AM Hi @Larry White yesterday's session 100804 is also missing log_sensor_file_ids for all video files including flir-avi, iphone-json/mov, and intel-bag files. Not sure what happened since during test session all files and sensor file ids generated fine.

Siddharth Patel 9:44 AM To be clear, only the sensor file ids are missing in the database - the actual files exist and are fine.

Siddharth Patel Today at 12:22 PM If you run the following query:

select * from log_sensor_file
where file_start_time > '2024-12-02 14:30:18' and file_start_time < '2024-12-02 14:30:23'

You get log_sensor_file table entries for yesterday's session 100804 task saccades_vertical. If you scroll down to the line for device iPhone - you see that the sensor_file_path array only has a single item in it - the hdf5 file. This array should have had the filepaths for iphone json and mov files. Because the system generated those files and they exist. Had those filepaths existed they would have had the same log_sensor_file_d (PK) as the hdf5 file. This is what I meant when I said that the log_sensor_file_ids for iphone json/mov files are missing. The log_sensor_file_id is missing because these filepaths are missing in the sensor_file_path array. 12:23 Now compare the above with the following query:

select * from log_sensor_file
where file_start_time > '2024-12-02 14:50:34' and file_start_time < '2024-12-02 14:50:46'

You get log_sensor_file table entries for yesterday's session 100804 task passage_reading. If you scroll down to the line for device iPhone - you see that the sensor_file_path array has the hdf5 file, but also the iphone json and mov files. Therefore for passage reading tasks there are no missing entries for device iPhone - the log_sensor_file_ids exist for all 3 files, and all 3 files were actually generated by the session and exist in storage volumes. (edited)

[lwhite1]

Re-reading the above, it seems like there are three issues:

1. Files that don't exist, but do have an entry in the log_sensor_file table (e.g. calibration.edf)
2. Files that exist, but don't have an entry in the table (e.g. IPhone movie files)
3. The mysterious 'test' file, which is generated in the calibration task

WRT #3, 'test' is the default value for the run() function in eyetracker_calibrate.py, so this is being called without a filename for some reason

Reviewing the database entries for the 12/2/24 session referenced above, there are 6 tasks that have entries for IPhone movie files (MOT, Passage, hevelius, calibration, DSC, and FingerNose). Of these six tasks, splitXDF is deferred for the first three (and only the first three as they produce large files). For all other tasks, splitXDF is performed during the session in a thread.

On 12/3, there was one (abbreviated) session, which produced iPhone.mov entries for calibration, finger-nose, and passage. While this is a subset of the tasks that produced entries on the prior day, it does still appear somewhat random. Also, the notes for the session read:

Participant arrived 20 minutes late. First session had an iPhone Panic before getting to calibration. Second session ran smooth. Calibration and validation successful. Finished Hevelius, then ended session due to time constraints.

Since Hevelius comes before passage and finger-nose, IDK why we have entries for those tasks.