As of v0.1.0, there remain several problems with WebNWB's file writing solution.
Large (1MB+) files encounter a memory overload error from h5wasm
We cannot update existing properties on h5wasm, so hdf5-io is forced to completely recompile the object into an HDF5 file regardless of the number of changes to that file.
Accessing file properties requires them to go through many (likely unnecessary) preprocessing steps before they are provided to the user. This results in very slow writing operations, such that a 1MB file may take several seconds to complete writing—or error as stated in (1).
Ideally, we can identify/negotiate a file access mode in h5wasm that allows updating existing properties. This would avoid (2) and (3) since properties are not accessed unless requested by the user. This may also clear up (1) since fewer h5wasm operations will bring file data into memory.
Isolated tests on the hdf5-io will likely be the best approach to fix (1) and (2). Notably, no tests—besides the initial fetching of a remote Ferguson file—are present on existing files like that. Such an implementation could uncover additional complications.
However, (3) is definitely an issue with WebNWB itself and/or esconform, which enforces the NWB Schema and the specification property provided in NWB files.
In summary, we should:
Ask the maintainer of h5wasm about updating existing file properties
Implement more write tests on existing NWB files in the hdf5-io repository.
Minimize the number of preprocessing steps triggered when object properties are accessed on NWBFile instances in WebNWB
Description of the bug
As of v0.1.0, there remain several problems with WebNWB's file writing solution.
h5wasm
h5wasm
, sohdf5-io
is forced to completely recompile the object into an HDF5 file regardless of the number of changes to that file.Ideally, we can identify/negotiate a file access mode in
h5wasm
that allows updating existing properties. This would avoid (2) and (3) since properties are not accessed unless requested by the user. This may also clear up (1) since fewerh5wasm
operations will bring file data into memory.Isolated tests on the hdf5-io will likely be the best approach to fix (1) and (2). Notably, no tests—besides the initial fetching of a remote Ferguson file—are present on existing files like that. Such an implementation could uncover additional complications.
However, (3) is definitely an issue with WebNWB itself and/or esconform, which enforces the NWB Schema and the
specification
property provided in NWB files.In summary, we should:
h5wasm
about updating existing file propertieshdf5-io
repository.Steps To Reproduce
Additional Information
N/A for now