search

LorenFrankLab / spyglass

Neuroscience data analysis framework for reproducible research built by Loren Frank Lab at UCSF
https://lorenfranklab.github.io/spyglass/
MIT License
91 stars 41 forks source link

nwbfile insertion fails #419

Closed MichaelCoulter closed 1 year ago

MichaelCoulter commented 1 year ago

This is the same error I got a few months ago with fred. Today I tried inserting data from a different rat pippin, and got the same error. The error says it cant find the video for the epoch, but I am pretty sure the video file exists and is correctly specified in the yml file. Thanks 

Creating a copy of NWB file pippin20210518.nwb with link to raw ephys data: pippin20210518_.nwb
Populate Session...
/stelmo/nwb/raw/pippin20210518_.nwb
Institution...
Lab...
LabMember...
Subject...
DataAcquisitionDevice...
Inserted data acquisition devices ['dataacq_device0']
CameraDevice...
Inserted camera devices ['MEC_sleep_camera', 'MEC_run_camera']
Probe...
No conforming probe device metadata found.
Skipping Apparatus for now...
IntervalList...
Populate ExperimenterList...
Populate ElectrodeGroup...
Populate Electrode...
Populate Raw...
Estimating sampling rate...
Estimated sampling rate: 30000.0
Importing raw data: Sampling rate:  30000.0 Hz
Number of valid intervals:  7
Populate SampleCount...
Populate DIOEvents...
Populate TaskEpochs
Populate StateScriptFile
Statescript log run 1
Statescript log run 2
Statescript log run 3
Statescript log run 1
Statescript log run 2
Statescript log run 3
Statescript log run 1
Statescript log run 2
Statescript log run 3
Statescript log run 1
Statescript log run 2
Statescript log run 3
Statescript log run 1
Statescript log run 2
Statescript log run 3
Statescript log run 1
Statescript log run 2
Statescript log run 3
Statescript log run 1
Statescript log run 2
Statescript log run 3
Populate VideoFile
No video found corresponding to epoch 01_s1
---------------------------------------------------------------------------
DuplicateError                            Traceback (most recent call last)
Cell In [24], line 27
     25 #raw_file_list = ['CH5_20210109.nwb', ]
     26 for file_name in raw_file_list:
---> 27     insert_sessions(file_name)

File ~/spyglass/src/spyglass/data_import/insert_sessions.py:46, in insert_sessions(nwb_file_names)
     44 copy_nwb_link_raw_ephys(nwb_file_name, out_nwb_file_name)
     45 Nwbfile().insert_from_relative_file_name(out_nwb_file_name)
---> 46 populate_all_common(out_nwb_file_name)

File ~/spyglass/src/spyglass/common/populate_all_common.py:46, in populate_all_common(nwb_file_name)
     44 StateScriptFile.populate(fp)
     45 print("Populate VideoFile")
---> 46 VideoFile.populate(fp)
     47 print("RawPosition...")
     48 PositionSource.insert_from_nwbfile(nwb_file_name)

File ~/anaconda3/envs/spyglass2/lib/python3.9/site-packages/datajoint/autopopulate.py:230, in AutoPopulate.populate(self, suppress_errors, return_exception_objects, reserve_jobs, order, limit, max_calls, display_progress, processes, make_kwargs, *restrictions)
    226 if processes == 1:
    227     for key in (
    228         tqdm(keys, desc=self.__class__.__name__) if display_progress else keys
    229     ):
--> 230         error = self._populate1(key, jobs, **populate_kwargs)
    231         if error is not None:
    232             error_list.append(error)

File ~/anaconda3/envs/spyglass2/lib/python3.9/site-packages/datajoint/autopopulate.py:281, in AutoPopulate._populate1(self, key, jobs, suppress_errors, return_exception_objects, make_kwargs)
    279 self.__class__._allow_insert = True
    280 try:
--> 281     make(dict(key), **(make_kwargs or {}))
    282 except (KeyboardInterrupt, SystemExit, Exception) as error:
    283     try:

File ~/spyglass/src/spyglass/common/common_behav.py:212, in VideoFile.make(self, key)
    207     if len(
    208         interval_list_contains(valid_times, video_obj.timestamps)
    209         > 0.9 * len(video_obj.timestamps)
    210     ):
    211         key["video_file_object_id"] = video_obj.object_id
--> 212         self.insert1(key)
    213         is_found = True
    215 if not is_found:

File ~/anaconda3/envs/spyglass2/lib/python3.9/site-packages/datajoint/table.py:334, in Table.insert1(self, row, **kwargs)
    327 def insert1(self, row, **kwargs):
    328     """
    329     Insert one data record into the table. For ``kwargs``, see ``insert()``.
    330 
    331     :param row: a numpy record, a dict-like object, or an ordered sequence to be inserted
    332         as one row.
    333     """
--> 334     self.insert((row,), **kwargs)

File ~/anaconda3/envs/spyglass2/lib/python3.9/site-packages/datajoint/table.py:443, in Table.insert(self, rows, replace, skip_duplicates, ignore_extra_fields, allow_direct_insert)
    439     raise err.suggest(
    440         "To ignore extra fields in insert, set ignore_extra_fields=True"
    441     )
    442 except DuplicateError as err:
--> 443     raise err.suggest(
    444         "To ignore duplicate entries in insert, set skip_duplicates=True"
    445     )

DuplicateError: ("Duplicate entry 'pippin20210518_.nwb-2-0' for key 'PRIMARY'", 'To ignore duplicate entries in insert, set skip_duplicates=True')
MichaelCoulter commented 1 year ago

I did not see any error messages print out during nwb file generation. Just the normal output: Checking associated file /cumulus/mcoulter/pippin/raw/20210518/20210518_pippin_02_r1.stateScriptLog Checking associated file /cumulus/mcoulter/pippin/raw/20210518/20210518_pippin_04_r2.stateScriptLog Checking associated file /cumulus/mcoulter/pippin/raw/20210518/20210518_pippin_06_r3.stateScriptLog in FlMdaExtractor: will write 6 files as a chunk

lfrank commented 1 year ago

I see a probe error as well, so you may want to look at that.

And can you post the yaml file?

On Mar 23, 2023, at 5:43 PM, MichaelCoulter @.***> wrote:

 This is the same error I got a few months ago with fred. Today I tried inserting data from a different rat pippin, and got the same error. The error says it cant find the video for the epoch, but I am pretty sure the video file exists and is ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd

This is the same error I got a few months ago with fred. Today I tried inserting data from a different rat pippin, and got the same error. The error says it cant find the video for the epoch, but I am pretty sure the video file exists and is correctly specified in the yml file. Thanks Creating a copy of NWB file pippin20210518.nwb with link to raw ephys data: pippin20210518.nwb Populate Session... /stelmo/nwb/raw/pippin20210518.nwb Institution... Lab... LabMember... Subject... DataAcquisitionDevice... Inserted data acquisition devices ['dataacq_device0'] CameraDevice... Inserted camera devices ['MEC_sleep_camera', 'MEC_run_camera'] Probe... No conforming probe device metadata found. Skipping Apparatus for now... IntervalList... Populate ExperimenterList... Populate ElectrodeGroup... Populate Electrode... Populate Raw... Estimating sampling rate... Estimated sampling rate: 30000.0 Importing raw data: Sampling rate: 30000.0 Hz Number of valid intervals: 7 Populate SampleCount... Populate DIOEvents... Populate TaskEpochs Populate StateScriptFile Statescript log run 1 Statescript log run 2 Statescript log run 3 Statescript log run 1 Statescript log run 2 Statescript log run 3 Statescript log run 1 Statescript log run 2 Statescript log run 3 Statescript log run 1 Statescript log run 2 Statescript log run 3 Statescript log run 1 Statescript log run 2 Statescript log run 3 Statescript log run 1 Statescript log run 2 Statescript log run 3 Statescript log run 1 Statescript log run 2 Statescript log run 3 Populate VideoFile No video found corresponding to epoch 01_s1

DuplicateError Traceback (most recent call last) Cell In [24], line 27 25 #raw_file_list = ['CH5_20210109.nwb', ] 26 for file_name in raw_file_list: ---> 27 insert_sessions(file_name)

File ~/spyglass/src/spyglass/data_import/insert_sessions.py:46, in insert_sessions(nwb_file_names) 44 copy_nwb_link_raw_ephys(nwb_file_name, out_nwb_file_name) 45 Nwbfile().insert_from_relative_file_name(out_nwb_file_name) ---> 46 populate_all_common(out_nwb_file_name)

File ~/spyglass/src/spyglass/common/populate_all_common.py:46, in populate_all_common(nwb_file_name) 44 StateScriptFile.populate(fp) 45 print("Populate VideoFile") ---> 46 VideoFile.populate(fp) 47 print("RawPosition...") 48 PositionSource.insert_from_nwbfile(nwb_file_name)

File ~/anaconda3/envs/spyglass2/lib/python3.9/site-packages/datajoint/autopopulate.py:230, in AutoPopulate.populate(self, suppress_errors, return_exception_objects, reserve_jobs, order, limit, max_calls, display_progress, processes, make_kwargs, *restrictions) 226 if processes == 1: 227 for key in ( 228 tqdm(keys, desc=self.class.name) if display_progress else keys 229 ): --> 230 error = self._populate1(key, jobs, **populate_kwargs) 231 if error is not None: 232 error_list.append(error)

File ~/anaconda3/envs/spyglass2/lib/python3.9/site-packages/datajoint/autopopulate.py:281, in AutoPopulate._populate1(self, key, jobs, suppress_errors, return_exception_objects, make_kwargs) 279 self.class._allow_insert = True 280 try: --> 281 make(dict(key), **(make_kwargs or {})) 282 except (KeyboardInterrupt, SystemExit, Exception) as error: 283 try:

File ~/spyglass/src/spyglass/common/common_behav.py:212, in VideoFile.make(self, key) 207 if len( 208 interval_list_contains(valid_times, video_obj.timestamps) 209 > 0.9 * len(video_obj.timestamps) 210 ): 211 key["video_file_object_id"] = video_obj.object_id --> 212 self.insert1(key) 213 is_found = True 215 if not is_found:

File ~/anaconda3/envs/spyglass2/lib/python3.9/site-packages/datajoint/table.py:334, in Table.insert1(self, row, kwargs) 327 def insert1(self, row, kwargs): 328 """ 329 Insert one data record into the table. For kwargs, see insert(). 330 331 :param row: a numpy record, a dict-like object, or an ordered sequence to be inserted 332 as one row. 333 """ --> 334 self.insert((row,), **kwargs)

File ~/anaconda3/envs/spyglass2/lib/python3.9/site-packages/datajoint/table.py:443, in Table.insert(self, rows, replace, skip_duplicates, ignore_extra_fields, allow_direct_insert) 439 raise err.suggest( 440 "To ignore extra fields in insert, set ignore_extra_fields=True" 441 ) 442 except DuplicateError as err: --> 443 raise err.suggest( 444 "To ignore duplicate entries in insert, set skip_duplicates=True" 445 )

DuplicateError: ("Duplicate entry 'pippin20210518_.nwb-2-0' for key 'PRIMARY'", 'To ignore duplicate entries in insert, set skip_duplicates=True')

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/LorenFrankLab/spyglass/issues/419__;!!LQC6Cpwp!puQoCWS2EeSbBDUowOmed_1PGdreDkpjXVdt9JKdrtXOM4RZ0Vu0D-6l0bBnX5T0_AWXGAwdoI0mMdBFKsSuK9UOHg$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ABV4PSLFQJUBB7PJMVNSUITW5TUZJANCNFSM6AAAAAAWF47FSY__;!!LQC6Cpwp!puQoCWS2EeSbBDUowOmed_1PGdreDkpjXVdt9JKdrtXOM4RZ0Vu0D-6l0bBnX5T0_AWXGAwdoI0mMdBFKsShsnPKcA$. You are receiving this because you are subscribed to this thread.Message ID: @.***>

MichaelCoulter commented 1 year ago

github wont let me add my file saying it does not support that data type. i will post it on slack

edeno commented 1 year ago

Can you use %debug to look at the error specifically what the values of the variables are in that function? For example, what are the values of the key dictionary?

samuelbray32 commented 1 year ago

Talked to Mike and he reports that this was solved by a change by @edeno in rec_to_nwb