search

BouchardLab / process_nwb

Functions for preprocessing timeseries data stored in the NWB format
https://process-nwb.readthedocs.io/en/latest/
4 stars 6 forks source link

Prepocessing module doesn't exisit but it does?!? #67

Closed jthermiz closed 2 years ago

jthermiz commented 2 years ago

I have a file /clusterfs/NSDS_data/nwb/RJH13/RJH13_B03.nwb that does not have a preprocessing module based on the readout from h5dump but when I go to preprocess it, I get this error:

(nsds_nwb) [jhermiz@n0000 scripts]$ python preprocess_folder /clusterfs/NSDS_data/jhermiz/nwb/RJH13/process
Processing /clusterfs/NSDS_data/jhermiz/nwb/RJH13/process/RJH13_B03.nwb

Traceback (most recent call last):
  File "preprocess_folder", line 54, in <module>
    all_steps=all_steps)
  File "/home/jhermiz/software/process_nwb/process_nwb/pipelines.py", line 124, in preprocess_block
    post_resample_rate=final_resample_rate)
  File "/home/jhermiz/software/process_nwb/process_nwb/wavelet_transform.py", line 419, in store_wavelet_transform
    processing.add(es)
  File "/home/jhermiz/.conda/envs/nsds_nwb/lib/python3.7/site-packages/hdmf/utils.py", line 580, in func_call
    return func(args[0], **pargs)
  File "/home/jhermiz/.conda/envs/nsds_nwb/lib/python3.7/site-packages/hdmf/container.py", line 724, in _func
    raise ValueError(msg)
ValueError: 'wvlt_amp_CAR_ln_downsampled_ECoG' already exists in ProcessingModule 'preprocessing'

Also when I try to remove the preprocessing module, using copy_strip I get a message that says it does not exist.

JesseLivezey commented 2 years ago

Was preprocessing ever run on that file previously? It seems possible that pynwb is storing some information in the nwb file when the preprocessing module is added that copy_strip isn't removing.

jthermiz commented 2 years ago

I just tried to create one from scratch using nsds_lab_to_nwb and I still get the same error. I'm attaching the log file generated by generate_nwb.py

RJH13_B03_log.txt

JesseLivezey commented 2 years ago

It looks like it is happening twice for some reason

INFO:nsds_lab_to_nwb.convert_block:Resampling...
INFO:nsds_lab_to_nwb.convert_block:Filtering and re-referencing...
INFO:nsds_lab_to_nwb.convert_block:Running wavelet transform...
INFO:nsds_lab_to_nwb.convert_block:Resampling...
INFO:nsds_lab_to_nwb.convert_block:Filtering and re-referencing...
INFO:nsds_lab_to_nwb.convert_block:Running wavelet transform...
jthermiz commented 2 years ago

Good call. It looks like there is duplicate code for resampling, filtering and transforming in the version that I was running. However, the latest version has another issue when all_steps=False. I will close this issue and open a new open for that.