search

KullmannLab / pyecog2

PyEcog2 is a python software package aimed at exploring, visualizing and analyzing (video) EEG telemetry data
GNU General Public License v3.0
4 stars 0 forks source link

Feature request #24

Closed mikailweston closed 3 years ago

mikailweston commented 3 years ago

When converting NDF to H5: Ability to load a csv file with animal ID and channel IDs

I have 32 animals in 4 rigs and need to convert NDFs iteratively as they are required. Each time I load the NDF converter I need to manually input animal ID, TIDs and date/time range.

either ability to save and load each conversion session to a csv file, or somehow save this information within the project file

mfpleite commented 3 years ago

Yup makes total sense. I'll save the NDF converter settings in the project file, so that the next time you need to convert files, only the dates need to be modified.

mfpleite commented 3 years ago

Done

mikailweston commented 3 years ago

Error when I try to reload saved NDF converter settings:

I created this project file then tried to convert an NDF folder with it, giving the error below for every single file. RIVC_Right_hM4Di.pyecog.txt

***
Progress: |--------------------------------------------------| 0.0% CompleteBuilding File Tree...

convert_ndf tids: [91, 92] 

Building File Tree...

convert_ndf tids: [91, 92] 

Building File Tree...
Traceback (most recent call last):
  File "C:\Users\mweston\Documents\GitHub\pyecog2\pyecog2\ndf_converter.py", line 645, in convert_ndf
    ndf = NdfFile(filename, fs = fs, verbose = False)
  File "C:\Users\mweston\Documents\GitHub\pyecog2\pyecog2\ndf_converter.py", line 113, in __init__
    self.read_file_metadata()
  File "C:\Users\mweston\Documents\GitHub\pyecog2\pyecog2\ndf_converter.py", line 132, in read_file_metadata
    assert (self.identifier == b' ndf')
AssertionError
Traceback (most recent call last):
  File "C:\Users\mweston\Documents\GitHub\pyecog2\pyecog2\ndf_converter.py", line 645, in convert_ndf
    ndf = NdfFile(filename, fs = fs, verbose = False)
  File "C:\Users\mweston\Documents\GitHub\pyecog2\pyecog2\ndf_converter.py", line 113, in __init__
    self.read_file_metadata()
  File "C:\Users\mweston\Documents\GitHub\pyecog2\pyecog2\ndf_converter.py", line 132, in read_file_metadata
    assert (self.identifier == b' ndf')
AssertionError
mfpleite commented 3 years ago

This is very old code... when interpreting the NDF file header jonny expects the first 4 bytes of the file to be b' ndf', then there is some metadata with the actual location of the start of the data packets received from the transmitters, which is the important part to get right at this stage.

The first couple of files in the folder that you sent: M1623324931.ndf and M1623328590.ndf do start with the b' ndf' identifier and are read correctly in my computer at least. But the last couple of files: M1623515030.ndf and M1623518680.ndf start with four zero bytes (b'\x00\x00\x00\x00') which is not the expected format of the NDFs. Something went wrong with the creation of those files. I can try to recover them, but this doesn't seem to be a bug in PyEcog

mikailweston commented 3 years ago

Ok, I suspect there was a data corruption on uploading or downloading the data. I'll try download it again. Thanks for looking into it.

On Mon, 14 Jun 2021, 10:36 mfpleite, @.***> wrote:

This is very old code... when interpreting the NDF file header jonny expects the first 4 bytes of the file to be b' ndf', then there is some metadata with the actual location of the start of the data packets received from the transmitters, which is the important part to get right at this stage.

The first couple of files in the folder that you sent: M1623324931.ndf and M1623328590.ndf do start with the b' ndf' identifier and are read correctly in my computer at least. But the last couple of files: M1623515030.ndf and M1623518680.ndf start with four zero bytes (b'\x00\x00\x00\x00') which is not the expected format of the NDFs. Something went wrong with the creation of those files. I can try to recover them, but this doesn't seem to be a bug in PyEcog

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/KullmannLab/pyecog2/issues/24#issuecomment-860544801, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB342ONEKKWKA3GNROO2JIDTSXEQPANCNFSM45YJNDOQ .

mfpleite commented 3 years ago

yeah they are empty files.

mikailweston commented 3 years ago

I found the source of the error: the back up drive was full so the upload wasn't completed.

On Mon, 14 Jun 2021 at 10:50, mfpleite @.***> wrote:

Closed #24 https://github.com/KullmannLab/pyecog2/issues/24.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/KullmannLab/pyecog2/issues/24#event-4884614123, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB342OKOBMLAVA3MPMUGQQDTSXGGRANCNFSM45YJNDOQ .