search

LorenFrankLab / spyglass

Neuroscience data analysis framework for reproducible research built by Loren Frank Lab at UCSF
https://lorenfranklab.github.io/spyglass/
MIT License
94 stars 43 forks source link

Cautious delete logging error #818

Closed edeno closed 9 months ago

edeno commented 9 months ago

Describe the bug

row = {'duration': 4.0610480308532715, 'dj_user': 'edeno', 'origin': '`decoding_clusterless_v1`.`clusterless_decoding_selection`', 'restriction': '[\'(`position_group_name`="test_group") AND (`nwb_file_name`="j1620210710_.nwb") AND (`waveform_features_group_name`="test_group") AND (`estimate_decoding_params`=True) AND (`encoding_interval`="06_r3 noPrePostTrialTimes") AND (`decoding_param_name`="j1620', 'merge_deletes': {}}

def _log_use(self, start, merge_deletes=None):
    365     """Log use of cautious_delete."""
--> 366     self._usage_table.insert1(
    367         dict(
    368             duration=time() - start,
    369             dj_user=dj.config["database.user"],
    370             origin=self.full_table_name,
    371             restriction=(
    372                 str(self.restriction)[:255] if self.restriction else "None"
    373             ),
    374             merge_deletes=merge_deletes,
    375         )
    376     )
DataError: (1406, "Data too long for column 'restriction' at row 1")

restriction evalutes to:

'[\'(`position_group_name`="test_group") AND (`nwb_file_name`="j1620210710_.nwb") AND (`waveform_features_group_name`="test_group") AND (`estimate_decoding_params`=True) AND (`encoding_interval`="06_r3 noPrePostTrialTimes") AND (`decoding_param_name`="j1620'
CBroz1 commented 9 months ago

Hmm... I thought I truncated the string, but maybe that needs to be 254 instead of 255

CBroz1 commented 9 months ago

Another reported error with the same method:

Cannot pack QueryExpression in blob Code ```python from spyglass.common import Nwbfile (Nwbfile & {'nwb_file_name' : 'RN2_20191110_.nwb'}).delete() ``` Error stack ```python [17:29:56][WARNING] Spyglass: Could not find lab team associated with Nwbfile. Be careful not to delete others' data. [17:30:00][INFO] Spyglass: Building merge cache for nwbfile. Found 6 downstream merge tables [2024-02-05 17:30:00,832][INFO]: Merge: Deleting 1 rows from `lfp_merge`.`l_f_p_output` [2024-02-05 17:30:00,890][INFO]: Deleting 3 rows from `lfp_band_v1`.`__l_f_p_band_v1` [2024-02-05 17:30:00,937][INFO]: Deleting 66 rows from `lfp_band_v1`.`l_f_p_band_selection__l_f_p_band_electrode` [2024-02-05 17:30:00,956][INFO]: Deleting 3 rows from `lfp_band_v1`.`l_f_p_band_selection` [2024-02-05 17:30:00,989][INFO]: Deleting 1 rows from `lfp_merge`.`l_f_p_output__l_f_p_v1` [2024-02-05 17:30:01,008][INFO]: Deleting 1 rows from `lfp_merge`.`l_f_p_output` Commit deletes? [yes, No]: yes [2024-02-05 17:30:03,431][INFO]: Deletes committed. [2024-02-05 17:30:03,486][INFO]: Deleting 1 rows from `lfp_v1`.`__l_f_p_v1` [2024-02-05 17:30:03,511][INFO]: Deleting 4 rows from `common_nwbfile`.`analysis_nwbfile` [2024-02-05 17:30:03,578][INFO]: Deleting 3 rows from `common_behav`.`_raw_position__pos_object` [2024-02-05 17:30:03,596][INFO]: Deleting 3 rows from `common_behav`.`_raw_position` [2024-02-05 17:30:03,628][INFO]: Deleting 3 rows from `common_behav`.`position_source__spatial_series` [2024-02-05 17:30:03,646][INFO]: Deleting 3 rows from `common_behav`.`position_source` [2024-02-05 17:30:03,679][INFO]: Deleting 36 rows from `common_dio`.`_d_i_o_events` [2024-02-05 17:30:03,737][INFO]: Deleting 22 rows from `lfp_electrode`.`l_f_p_electrode_group__l_f_p_electrode` [2024-02-05 17:30:03,763][INFO]: Deleting 88 rows from `common_ephys`.`_electrode` [2024-02-05 17:30:03,783][INFO]: Deleting 22 rows from `common_ephys`.`_electrode_group` [2024-02-05 17:30:03,815][INFO]: Deleting 1 rows from `common_ephys`.`_raw` [2024-02-05 17:30:03,869][INFO]: Deleting 1 rows from `common_ephys`.`_sample_count` [2024-02-05 17:30:03,921][INFO]: Deleting 3 rows from `common_behav`.`__position_interval_map` [2024-02-05 17:30:03,968][INFO]: Deleting 3 rows from `common_behav`.`_state_script_file` [2024-02-05 17:30:03,999][INFO]: Deleting 3 rows from `common_behav`.`_video_file` [2024-02-05 17:30:04,018][INFO]: Deleting 3 rows from `common_task`.`_task_epoch` [2024-02-05 17:30:04,056][INFO]: Deleting 1 rows from `lfp_v1`.`l_f_p_selection` [2024-02-05 17:30:04,078][INFO]: Deleting 9 rows from `common_interval`.`interval_list` [2024-02-05 17:30:04,108][INFO]: Deleting 1 rows from `common_session`.`_session__data_acquisition_device` [2024-02-05 17:30:04,138][INFO]: Deleting 2 rows from `common_session`.`_session__experimenter` [2024-02-05 17:30:04,169][INFO]: Deleting 1 rows from `lfp_electrode`.`l_f_p_electrode_group` [2024-02-05 17:30:04,189][INFO]: Deleting 1 rows from `common_session`.`_session` [2024-02-05 17:30:04,208][INFO]: Deleting 1 rows from `common_nwbfile`.`nwbfile` Commit deletes? [yes, No]: yes [2024-02-05 17:30:05,355][INFO]: Deletes committed. --------------------------------------------------------------------------- DataJointError Traceback (most recent call last) Cell In[9], line 2 1 from spyglass.common import Nwbfile ----> 2 (Nwbfile & {'nwb_file_name' : 'RN2_20191108_.nwb'}).delete() File ~/Src/spyglass/src/spyglass/utils/dj_mixin.py:437, in SpyglassMixin.delete(self, force_permission, *args, **kwargs) 435 def delete(self, force_permission=False, *args, **kwargs): 436 """Alias for cautious_delete, overwrites datajoint.table.Table.delete""" --> 437 self.cautious_delete(force_permission=force_permission, *args, **kwargs) File ~/Src/spyglass/src/spyglass/utils/dj_mixin.py:429, in SpyglassMixin.cautious_delete(self, force_permission, *args, **kwargs) 425 return 427 super().delete(*args, **kwargs) # Additional confirm here --> 429 self._log_use(start=start, merge_deletes=merge_deletes) File ~/Src/spyglass/src/spyglass/utils/dj_mixin.py:366, in SpyglassMixin._log_use(self, start, merge_deletes) 364 def _log_use(self, start, merge_deletes=None): 365 """Log use of cautious_delete.""" --> 366 self._usage_table.insert1( 367 dict( 368 duration=time() - start, 369 dj_user=dj.config["database.user"], 370 origin=self.full_table_name, 371 restriction=( 372 str(self.restriction)[:255] if self.restriction else "None" 373 ), 374 merge_deletes=merge_deletes, 375 ) 376 ) File ~/miniconda3/envs/spyglass-env/lib/python3.9/site-packages/datajoint/table.py:337, in Table.insert1(self, row, **kwargs) 330 def insert1(self, row, **kwargs): 331 """ 332 Insert one data record into the table. For ``kwargs``, see ``insert()``. 333 334 :param row: a numpy record, a dict-like object, or an ordered sequence to be inserted 335 as one row. 336 """ --> 337 self.insert((row,), **kwargs) File ~/miniconda3/envs/spyglass-env/lib/python3.9/site-packages/datajoint/table.py:419, in Table.insert(self, rows, replace, skip_duplicates, ignore_extra_fields, allow_direct_insert) 416 return 418 field_list = [] # collects the field list from first row (passed by reference) --> 419 rows = list( 420 self.__make_row_to_insert(row, field_list, ignore_extra_fields) 421 for row in rows 422 ) 423 if rows: 424 try: File ~/miniconda3/envs/spyglass-env/lib/python3.9/site-packages/datajoint/table.py:420, in (.0) 416 return 418 field_list = [] # collects the field list from first row (passed by reference) 419 rows = list( --> 420 self.__make_row_to_insert(row, field_list, ignore_extra_fields) 421 for row in rows 422 ) 423 if rows: 424 try: File ~/miniconda3/envs/spyglass-env/lib/python3.9/site-packages/datajoint/table.py:872, in Table.__make_row_to_insert(self, row, field_list, ignore_extra_fields) 870 elif isinstance(row, collections.abc.Mapping): # dict-based 871 check_fields(row) --> 872 attributes = [ 873 self.__make_placeholder(name, row[name], ignore_extra_fields) 874 for name in self.heading 875 if name in row 876 ] 877 else: # positional 878 try: File ~/miniconda3/envs/spyglass-env/lib/python3.9/site-packages/datajoint/table.py:873, in (.0) 870 elif isinstance(row, collections.abc.Mapping): # dict-based 871 check_fields(row) 872 attributes = [ --> 873 self.__make_placeholder(name, row[name], ignore_extra_fields) 874 for name in self.heading 875 if name in row 876 ] 877 else: # positional 878 try: File ~/miniconda3/envs/spyglass-env/lib/python3.9/site-packages/datajoint/table.py:809, in Table.__make_placeholder(self, name, value, ignore_extra_fields) 807 value = value.bytes 808 elif attr.is_blob: --> 809 value = blob.pack(value) 810 value = ( 811 self.external[attr.store].put(value).bytes 812 if attr.is_external 813 else value 814 ) 815 elif attr.is_attachment: File ~/miniconda3/envs/spyglass-env/lib/python3.9/site-packages/datajoint/blob.py:594, in pack(obj, compress) 590 assert isinstance(obj, bytes) and obj.startswith( 591 (b"ZL123\0", b"mYm\0", b"dj0\0") 592 ) 593 return obj --> 594 return Blob().pack(obj, compress=compress) File ~/miniconda3/envs/spyglass-env/lib/python3.9/site-packages/datajoint/blob.py:576, in Blob.pack(self, obj, compress) 574 def pack(self, obj, compress): 575 self.protocol = b"mYm\0" # will be replaced with dj0 if new features are used --> 576 blob = self.pack_blob( 577 obj 578 ) # this may reset the protocol and must precede protocol evaluation 579 blob = self.protocol + blob 580 if compress and len(blob) > 1000: File ~/miniconda3/envs/spyglass-env/lib/python3.9/site-packages/datajoint/blob.py:204, in Blob.pack_blob(self, obj) 202 return self.pack_uuid(obj) 203 if isinstance(obj, collections.abc.Mapping): --> 204 return self.pack_dict(obj) 205 if isinstance(obj, str): 206 return self.pack_string(obj) File ~/miniconda3/envs/spyglass-env/lib/python3.9/site-packages/datajoint/blob.py:445, in Blob.pack_dict(self, d) 441 def pack_dict(self, d): 442 return ( 443 b"\4" 444 + len_u64(d) --> 445 + b"".join( 446 b"".join((len_u64(it) + it) for it in packed) 447 for packed in (map(self.pack_blob, pair) for pair in d.items()) 448 ) 449 ) File ~/miniconda3/envs/spyglass-env/lib/python3.9/site-packages/datajoint/blob.py:446, in (.0) 441 def pack_dict(self, d): 442 return ( 443 b"\4" 444 + len_u64(d) 445 + b"".join( --> 446 b"".join((len_u64(it) + it) for it in packed) 447 for packed in (map(self.pack_blob, pair) for pair in d.items()) 448 ) 449 ) File ~/miniconda3/envs/spyglass-env/lib/python3.9/site-packages/datajoint/blob.py:446, in (.0) 441 def pack_dict(self, d): 442 return ( 443 b"\4" 444 + len_u64(d) 445 + b"".join( --> 446 b"".join((len_u64(it) + it) for it in packed) 447 for packed in (map(self.pack_blob, pair) for pair in d.items()) 448 ) 449 ) File ~/miniconda3/envs/spyglass-env/lib/python3.9/site-packages/datajoint/blob.py:210, in Blob.pack_blob(self, obj) 208 return self.pack_bytes(obj) 209 if isinstance(obj, collections.abc.MutableSequence): --> 210 return self.pack_list(obj) 211 if isinstance(obj, collections.abc.Sequence): 212 return self.pack_tuple(obj) File ~/miniconda3/envs/spyglass-env/lib/python3.9/site-packages/datajoint/blob.py:422, in Blob.pack_list(self, t) 418 def pack_list(self, t): 419 return ( 420 b"\2" 421 + len_u64(t) --> 422 + b"".join(len_u64(it) + it for it in (self.pack_blob(i) for i in t)) 423 ) File ~/miniconda3/envs/spyglass-env/lib/python3.9/site-packages/datajoint/blob.py:422, in (.0) 418 def pack_list(self, t): 419 return ( 420 b"\2" 421 + len_u64(t) --> 422 + b"".join(len_u64(it) + it for it in (self.pack_blob(i) for i in t)) 423 ) File ~/miniconda3/envs/spyglass-env/lib/python3.9/site-packages/datajoint/blob.py:422, in (.0) 418 def pack_list(self, t): 419 return ( 420 b"\2" 421 + len_u64(t) --> 422 + b"".join(len_u64(it) + it for it in (self.pack_blob(i) for i in t)) 423 ) File ~/miniconda3/envs/spyglass-env/lib/python3.9/site-packages/datajoint/blob.py:217, in Blob.pack_blob(self, obj) 215 if obj is None: 216 return self.pack_none() --> 217 raise DataJointError( 218 "Packing object of type %s currently not supported!" % type(obj) 219 ) DataJointError: Packing object of type currently not supported! ```