Closed CodyCBakerPhD closed 1 month ago
This branch should work for HDF5 at least: https://github.com/hdmf-dev/hdmf/pull/1110
I don't think NeuroConv changes should be needed since they just dynamically pass everything down the chain
Just flagging that adding Paul's safe conversion strategy to https://github.com/NeurodataWithoutBorders/nwb-guide/pull/778 would change a good amount of our error handling for the GUIDE in general, and might be better in a separate Issue / PR
We can probably just use our own existing endpoint for per-file conversions; the main goal with any kind of 'safe' way of doing that is just that the traceback error stack gets dumped to a persistent file on disk since stdout/stderr pipes are not easily accessible during multiprocessing
Just implemented a general log system in https://github.com/NeurodataWithoutBorders/nwb-guide/pull/778 with a specific endpoint for registering errors from parallel processes! Should give us something useful to work with generally for remote debugging :)
Here's an example file: 2024-05-15_16-03-22.log
As shown by the Exception on /neuroconv/convert
line, this will automatically log errors from any failing endpoint in addition to the process-specific errors
Just implemented a general log system in https://github.com/NeurodataWithoutBorders/nwb-guide/pull/778 with a specific endpoint for registering errors from parallel processes! Should give us something useful to work with generally for remote debugging :)
Very cool!
Added in #778
What would you like to see added to the NWB GUIDE?
Next step after https://github.com/NeurodataWithoutBorders/nwb-guide/pull/676 is to do parallel file write and sub bars for the buffers of each file
Here is some basic code that should accomplish this (I'll have to open a small PR to HDMF and HDMF-Zarr to facilitate proper progress bar class specification)
where
safe_session_to_nwb
looks like thiswhere
session_to_nwb
is whatever current function we currently use in the GUIDE to convert a single session to create a single fileInspired by @pauladkisson's contribution on https://github.com/catalystneuro/cookiecutter-my-lab-to-nwb-template/pull/23, which is based on strategies we've used many times for past conversions
Do you have any interest in helping implement the feature?
Yes.
Code of Conduct
Yes
Did you confirm this feature was not already reported?
Yes