search

open-ephys / open-ephys-python-tools

Python code for interacting with the Open Ephys GUI
MIT License
29 stars 17 forks source link

Error loading continuous files: AssertionError assert numRecords % 1 == 0 #38

Open stef1029 opened 1 month ago

stef1029 commented 1 month ago

Hi,

I'm having another issue with loading some data I've recorded. The error I'm getting is this:

Traceback (most recent call last):
  File "/cephfs2/srogers/Behaviour code/2407_July_WT_cohort/Analysis/NAP/July_cohort_scripts/analysis_manager.py", line 36, in __init__
    self.ingest_behaviour_data()
  File "/cephfs2/srogers/Behaviour code/2407_July_WT_cohort/Analysis/NAP/July_cohort_scripts/analysis_manager.py", line 83, in ingest_behaviour_data
    self.OEAB_data = process_ADC_Recordings(self.OEAB_folder, self.rig_id)
  File "/cephfs2/srogers/Behaviour code/2407_July_WT_cohort/Analysis/NAP/July_cohort_scripts/Preliminary_analysis_scripts/process_ADC_recordings.py", line 28, in __init__
    self.extract_ADC_data()
  File "/cephfs2/srogers/Behaviour code/2407_July_WT_cohort/Analysis/NAP/July_cohort_scripts/Preliminary_analysis_scripts/process_ADC_recordings.py", line 35, in extract_ADC_data
    self.recording = open_ephys.analysis.Session(self.dirname).recordnodes[0].recordings[0].continuous[0]
  File "/lmb/home/srogers/anaconda3/envs/behaviour_analysis/lib/python3.10/site-packages/open_ephys/analysis/recording.py", line 81, in continuous
    self.load_continuous()
  File "/lmb/home/srogers/anaconda3/envs/behaviour_analysis/lib/python3.10/site-packages/open_ephys/analysis/formats/OpenEphysRecording.py", line 252, in load_continuous
    self._continuous.append(self.Continuous(stream_info[stream_index],
  File "/lmb/home/srogers/anaconda3/envs/behaviour_analysis/lib/python3.10/site-packages/open_ephys/analysis/formats/OpenEphysRecording.py", line 63, in __init__
    self._sample_numbers_internal, _, _, self.valid_records = load(files[0], recording_index)
  File "/lmb/home/srogers/anaconda3/envs/behaviour_analysis/lib/python3.10/site-packages/open_ephys/analysis/formats/helpers/oe_fast_loader.py", line 105, in load
    return load_continuous(filename, recording_index)
  File "/lmb/home/srogers/anaconda3/envs/behaviour_analysis/lib/python3.10/site-packages/open_ephys/analysis/formats/helpers/oe_fast_loader.py", line 152, in load_continuous
    numRecords = getNumRecords(filename, RECORD_SIZE)
  File "/lmb/home/srogers/anaconda3/envs/behaviour_analysis/lib/python3.10/site-packages/open_ephys/analysis/formats/helpers/oe_fast_loader.py", line 80, in getNumRecords
    assert numRecords % 1 == 0
AssertionError

I'm not sure why these files are having these issues regarding issues at the time of recording. I did, however, notice that a large number of my recording files had not saved the structure.openephys file at the end of recording. When i originally tried to load these files I got the error IndexError: list index out of range from the line self.recording = open_ephys.analysis.Session(self.dirname).recordnodes[0].recordings[0].continuous[0].

To try and fix this, I tried transplanting in structure files from healthy folders, however that is when I started getting this error. The contents of the structure files seems to be correct, unless bitVolts is the cause of the issue. Primarily I'm wondering if there's something I can do to fix this issue, and if there are any known reasons why the structure files failed to save in the first place. It's happened on a very large number of recordings over the last couple weeks, so I don't think anything was happening obviously indicating a failure at the ends of recordings as I would have noticed this number of crashes or something like that.

Thanks very much for any help you can give,

Stefan

medengineer commented 1 month ago

Hi Stefan,

The structure files are actually generated at the start of each recording. We haven't seen any cases where only the structure file is missing.

Did anything in your configuration change between the healthy recordings and those triggering this error? Are you using a script to start/stop recording? If so, can you send any scripts and/or one of these incomplete recordings to pavel@open-ephys.org?

medengineer commented 1 month ago

Update:

Tested and confirmed the structure file writing can get skipped if the GUI is quit before recording is finished. This can currently only happen if QUIT is called in quick succession after IDLE when using the HTTPServer.

The workaround for now is to ensure a sufficient time.sleep(N) is inserted between idle() and quit() calls when controlling the GUI via the HTTPServer. However this is imprecise and unenforceable.

As a robust solution, I will add a check in the GUI's HTTPServer quit() method to ensure all recording threads have finished prior to closing the GUI.