How to close the recording file? (MEArecRecordingExtractor)

[ZoeChen96]

Hi,

I wanted to modify the .h5 MEArec file by some customized preprocessing, so I have to overwrite the 'recordings' in .h5 MEArec file. And I found a problem: after I use MEArecRecordingExtractor for a file (say recording_Test.h5), I can't access to that file again. It shows the error that OSError: Unable to synchronously open file (file is already open for read-only). Is there any method to close the recording? You can find the details below:

The whole error looks like:

OSError Traceback (most recent call last) Cell In[5], line 1 ----> 1 with h5py.File('recording_Test.h5', 'r+') as record_h5: 2 record_h5['recordings'].write_direct(spike_data) File /imec/other/macaw/chen14/anaconda43/envs/si96_pure/lib/python3.9/site-packages/h5py/_hl/files.py:567, in File.init(self, name, mode, driver, libver, userblock_size, swmr, rdcc_nslots, rdcc_nbytes, rdcc_w0, track_order, fs_strategy, fs_persist, fs_threshold, fs_page_size, page_buf_size, min_meta_keep, min_raw_keep, locking, alignment_threshold, alignment_interval, meta_block_size, kwds) 558 fapl = make_fapl(driver, libver, rdcc_nslots, rdcc_nbytes, rdcc_w0, 559 locking, page_buf_size, min_meta_keep, min_raw_keep, 560 alignment_threshold=alignment_threshold, 561 alignment_interval=alignment_interval, 562 meta_block_size=meta_block_size, 563 kwds) 564 fcpl = make_fcpl(track_order=track_order, fs_strategy=fs_strategy, 565 fs_persist=fs_persist, fs_threshold=fs_threshold, 566 fs_page_size=fs_page_size) --> 567 fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr) 569 if isinstance(libver, tuple): 570 self._libver = libver File /imec/other/macaw/chen14/anaconda43/envs/si96_pure/lib/python3.9/site-packages/h5py/_hl/files.py:233, in make_fid(name, mode, userblock_size, fapl, fcpl, swmr) 231 fid = h5f.open(name, flags, fapl=fapl) 232 elif mode == 'r+': --> 233 fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl) 234 elif mode in ['w-', 'x']: 235 fid = h5f.create(name, h5f.ACC_EXCL, fapl=fapl, fcpl=fcpl) File h5py/_objects.pyx:54, in h5py._objects.with_phil.wrapper() File h5py/_objects.pyx:55, in h5py._objects.with_phil.wrapper() File h5py/h5f.pyx:106, in h5py.h5f.open() OSError: Unable to synchronously open file (file is already open for read-only)

[ZoeChen96]

BTW, I checked the .h5 file writing code, it can run for many times without giving errors.

[alejoe91]

Hi @ZoeChen96

You shouldn't save on the same file! What does spike_data contain? Can't you make another file instead?

[ZoeChen96]

Hi Alessio, I wanted to test if I can do this overwrite correctly or not... e.g. I generated 3 datasets with mearec, one is spike_data with spikes, one is noise_data, and another random thing which I called it Test_data (which I use as recording_Test.h5). And I want to verify whether I can overwrite the data correctly. So I tried to write spike_data 's trace inside recording_Test.h5, trying to get_traces() and compare with spike_data, and then I want to write noise_data trace inside to compare again for the whole verification. Actually, you are correct: I can make another file. But my first thought was to verify using the method above, so I can be sure that I can overwrite the file properly. But in actual use I will take care of different datasets. But is there any possibility to close the MEArecRecordingExtractor?

[alejoe91]

Hi @ZoeChen96

The file gets closed if you do del recording_manual but again you should NOT overwrite the file that way! MEArec has a complex API/file format itself, so you can check its documentation if you really want to overwrite a field of the h5 file.

I'll close this for now!

[ZoeChen96]

Hi Alessio, thanks for the information! I tried the del recording_manual before but the problem still exists. It seems recording_manual disappears, but the file opening is still somewhere else. Actually I already checked the structure of MEArec .h5 file, and that's the only method I can think of to do some customized preprocessing... 😟but maybe that's because of I am not very familiar with python. So I want to ask a little question here: can I do some customized preprocessing besides the available preprocessing?

• I tried to use the numpyrecording to create a new recording from numpy, and add the probe/freq information manually, like in the tutorial https://spikeinterface.readthedocs.io/en/latest/modules_gallery/core/plot_1_recording_extractor.html#sphx-glr-modules-gallery-core-plot-1-recording-extractor-py but I think the dumpable attribute is a problem there, as it requires me to save() which will lead me to a second copy of the dataset.
• I wonder if I can do straightly after the RecordingExtractor, like you did: get a RecordingExtractor in and return a RecordingExtractor. Do you think that's possible? And do you have any instruction for it?

[alejoe91]

@ZoeChen96 what kind of preprocessing do you need?

Anyways, the best approach is to use a NumpyRecording and then save() (you need to use n_jobs=1). That returns a new binary recording extractor. Note that the data is not duplicated: the NumpyRecording is in-memory (on your RAM), while the saved recording is dumped to a file.

An alternative method would be to write a spikeinterface.preprocessing class + function (see a simple example here), but this would require some more advanced coding skills.

[ZoeChen96]

@alejoe91 Hi Alessio, as for the preprocessing, I need to include some BCI recording-systems' non-ideality features (e.g. quantization, ...).
Anyway, your information is very useful (as usual), I will have a look at it. Thanks a lot!

[alejoe91]

Let me point out that you can use the adc_bit_depth, lsb, or gain to control quantization: https://github.com/alejoe91/MEArec/blob/master/MEArec/default_params/recordings_params.yaml#L55-L57

Sorry this is not well documented, I need to add all these new features to the MEArec docs! It should be relatively easy to understand what they do if you're familiar with the concept

[ZoeChen96]

Hi Alessio, that is sooo helpful! I did not realize that before. Thanks a lot!!!