search

LorenFrankLab / rec_to_nwb

Data Migration REC -> NWB 2.0 Service
Other
2 stars 8 forks source link

Order of .part files incorrect #37

Closed jguides closed 2 years ago

jguides commented 2 years ago

Generating preprocessing files for a recording with .part files (20201104_peanut_20_h2) seems to have failed from the .part files being appended out of order. This seems to have happened because part10 and part11 are viewed as earlier than part2, part3 etc. I was not able to find a section of code in rec_to_nwb where the sorting of .part files happens, so I suspect this may occur in rec_to_binaries. @edeno I am wondering if you could confirm this. Thanks in advance for any help with this.

(ID: 19) Appending from file: "/nimbus/jguidera/peanut/raw/20201104/20201104_peanut_20_h2.part10.rec" (ID: 19) Calculating time offset for appended file... (ID: 19) WARNING: this file does not contain information about the date and time when the file was created, so if the system clock was reset there is no way of knowing how much time elapsed between the two files. No offset will be applied. (ID: 19) 0 % (ID: 19) 5 % (ID: 19) 10 % (ID: 19) 15 % (ID: 19) 20 % (ID: 19) 25 % (ID: 19) 30 % (ID: 19) 35 % (ID: 19) 40 % (ID: 19) 45 % (ID: 19) 50 % (ID: 19) 55 % (ID: 19) 60 % (ID: 19) 65 % (ID: 19) Gap of 94491 found. Packet: 71081448 (ID: 19) 70 % (ID: 19) 75 % (ID: 19) 80 % (ID: 19) 85 % (ID: 19) 90 % (ID: 19) 95 % (ID: 19) (ID: 19) Appending from file: "/nimbus/jguidera/peanut/raw/20201104/20201104_peanut_20_h2.part11.rec" (ID: 19) Calculating time offset for appended file... (ID: 19) WARNING: this file does not contain information about the date and time when the file was created, so if the system clock was reset there is no way of knowing how much time elapsed between the two files. No offset will be applied. (ID: 19) (ID: 19) Appending from file: "/nimbus/jguidera/peanut/raw/20201104/20201104_peanut_20_h2.part2.rec" (ID: 19) Calculating time offset for appended file... (ID: 19) WARNING: this file does not contain information about the date and time when the file was created, so if the system clock was reset there is no way of knowing how much time elapsed between the two files. No offset will be applied. (ID: 19) Error: timestamps do not begin with greater value than the end of the last file. Aborting. (ID: 19) (ID: 19) Done (ID: 19) (ID: 19) Done (ID: 19) (ID: 19) Done

edeno commented 2 years ago

Hi @jguides,

It does happen in rec_to_binaries. I'll fix how the files are sorted.

edeno commented 2 years ago

Fixed by version 0.7.4 in rec_to_binaries