Closed VBaratham closed 2 years ago
@VBaratham, this seems like 2 issues. 1 is that we typically want to downsample the final amplitudes and 2 is that we should have more flexibility with what information gets saved.
Yeah, it is two separate issues. I think 1 sort of depends on 2 b/c it's not so helpful to downsample the final amplitudes if you still have to store all the intermediate datasets. In any case I'm hoping to implement them together.
On Fri, Mar 5, 2021 at 5:49 PM Jesse Livezey notifications@github.com wrote:
@VBaratham https://github.com/VBaratham, this seems like 2 issues. 1 is that we typically want to downsample the final amplitudes and 2 is that we should have more flexibility with what information gets saved.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/BouchardLab/process_nwb/issues/37#issuecomment-791835599, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHH3TAO3LI3DHYNEOHCDFTTCGCZPANCNFSM4YWDJLRA .
@VBaratham one problem with not writing (some of) the intermediate results, is that it will require things to be stored in memory at certain intervals during preprocessing. For instance, if the wavelet amplitudes are not stored, then they will have to be stored in memory to be resampled before being stored.
I suppose that we could create a joint pipeline that does multiple steps inside of an iterative write.
Which is only a problem for datasets that might not fit in memory.
3 potential pipelines are
Store all (used if someone is looking at data at different points in pipeline). Individual ops can be iterative, if needed.
first_resample
(store) (maybe initial_resample_rate
)final_resample
(store) (maybe final_resample_rate
)Store final (simple for datasets that fit in memory). Read from NWB, apply in numpy, write to NWB
first_resample
final_resample
(store)Store final (iterative write, for datasets that do not fit in memory). Read from NWB as needed to compute values.
first_resample
. Downsampled data must all fit in memory.2 and 3 would produce the same NWB file but could be selected based on memory limitations. 1 would produce a larger NWB file.
Yeah, I didn't realize how complicated the issue of first vs final resample makes this. I agree that those options all sound reasonable and I can't think of any other possibilities.
For 3, how about if we add "resample to final_resample
" as part of the wavelet step? In the current pipeline, I think they are the only 2 steps that need to be combined. It should be simple to do iteratively or all at once.
We can use --first-resample-rate
and --final-resample-rate
if you prefer the longer argument names
I'll get my iterative write PR cleaned up and try to take a crack at this, let me know what you think
Closed by #60
Currently this module performs the entire downsample operation before doing anything else (CAR, wavelet) which is problematic if you want to compute wavelet amplitudes for frequencies above 1/2 the downsample frequency. Our auditory analyses operate on frequency bands up to 1200 hz (center freq), so we typically downsample first to 3200 Hz, perform CAR and wavelet transform at 3200hz, then downsample again to 400 hz before saving the frequency decomposition.
Right now we have
apply_*
functions which perform one processing step on a numpy array, then we havestore_*
functions which callapply_*()
, create an NWB dataset, and write it into the NWB file. I think all we need is another function that creates and returns the dataset without putting it into the NWB file. Then theprocess_folder
script will call those functions instead ofstore_*()
, and save whichever datasets are needed itself. Process_folder will also take--first-resample
and--final-resample
as args. Also need an option to specify which datasets to store (or which not to store?)