Closed acomrie closed 2 years ago
I can't think of any reason that would cause a problem, so please do try to fix it.
I have attempted this, and unfortunately filenames are parsed and inferred in a bunch of different places, and the different places do so slightly differently (rather than all with a few core fxns). I don't think it will be smart to half-fix it with my current changes, and it would be better to either leave it as is and update our lab file naming documentation, or to overhaul some of the rec to nwb code to be more internally consistent in terms of filenaming expectations. I'll go with the former given time considerations at the moment.
fyi in case people come across this in the future, the updated documentation from google drive now starts w/:
Basic file name format basic regular expression to match file (python re format): ^(\d)_([a-zA-Z0-9])(\d*){0,1}(\w).{0,1}(.).([a-zA-Z0-9]*)$
6 groups: date, animal name, epoch #, label (optional), continuation extension (optional), file extension
date: although reg ex only matches digits, the date should be of format %YYYY%MM%DD animal name: alphanumeric epoch: digits (every epoch 2 digits ##, multiple epochs in a file by series to digit pairs 010203) label: alphanumeric (optional) continuation extension: anything (recommend separator as period '.' but can be any whitespace non-alphanumeric. If separator is period, it is excluded from the match, if it is anything else it is included in the match. For camera/position data used to signify file continuation e.g. 01, 02, 03....) file extension: alphanumeric (e.g. rec, stateScript, h264)
20220506 UPDATE: our current rec_to_nwb code does NOT use the regular expression described above Instead use this format: date_animalname_epochnumber_label.fileextension With date as digits in format %YYYY%MM%DD, animalname as alphanumeric and the SAME as the animal directory name, epoch number as 2 digits (for ex: 01, 02, ... 13, 14...), and the final field before the .fileextension should describe the epoch type alphanumerically without any more underscores or periods (for ex: sleep, run, sleep5, r5, linear9...).
It appears that
dataset_names
is determined in two slightly different ways, but then the one way is used to access the other. Most of the time the two ways yield the same result, but when they do not, a bug arises.nwb_file_builder.py
's__extract_datasets()
callsself.data_scanner.extract_data_from_date_folder()
, which goes on to use__extract_experiments()
which then uses__extract_datasets()
indata_scanner.py
. That final__extract_datasets()
determinesdataset_names
with one approach. Then, separately, innwb_file_builder.py
,dataset_names
is determined with another approach, usingself.data_scanner.get_all_epochs()
. It seems like these two spots should use the same rather than two slightly different approaches to determinedataset_names
. That is because later, innwb_file_builder.py
, we do the following list comprehension:[self.data_scanner.data[animal_name][date][dataset] for dataset in self.dataset_name]
If the two approaches have yielded distinctdataset_names
, then thedataset
key will not have any value.The two approaches seem to yield distinct
dataset_names
when a filename has an optional field following the sleep/run descriptor. For example, instead of _date_animal_epochnumberdescription.fileending, if the filename is _date_animal_epochnumber_descriptionoptionaltag.fileending, then thedataset_names
may have either the form _epochnumberoptionaltag or _epochnumberdescription with the two appraoches, respectively. Instead, we want both methods to yield a string of the form _epochnumberdescription.I can rename files to avoid having this optionaltag at the end of the filename if needed, but it would be useful to be able to add optional info into the filename, while still satisfying our documentation's naming conventions (per regular expression:
^(\d*)_([a-zA-Z0-9]*)_(\d*)_{0,1}(\w*)\.{0,1}(.*)\.([a-zA-Z0-9]*)$
). Moreover, it is still odd for us to determinedataset_names
in two different ways and then use them as if they are the exact same. Is there a reason it is the way it is that I am missing? Would it be okay to change it so that bothget_all_epochs()
and__extract_datasets()
determine the dataset_names the same way? I can work on PR if someone who is more familiar with this code thinks it seems okay, but I am not super familiar with this code so wanted to see what folks think first. Thanks much for any thoughts.Example error -
And then if we take a look here,
self.data_scanner.data[animal_name][date]
evaluates toSo it is looking for
'01_sleep'
correctly, but the thing it is looking for is currently incorrectly called'sleep_p1'