LorenFrankLab / spyglass

Neuroscience data analysis framework for reproducible research built by Loren Frank Lab at UCSF
https://lorenfranklab.github.io/spyglass/
MIT License
94 stars 43 forks source link

Delete seems to have left some residual trace. Cannot reinsert data when old data is deleted. #1166

Closed shijiegu closed 1 month ago

shijiegu commented 1 month ago

Describe the bug sdi.insert_sessions('lewis20240115.nwb') gives

---------------------------------------------------------------------------
DataJointError                            Traceback (most recent call last)
Cell In[36], line 1
----> 1 sdi.insert_sessions('lewis20240115.nwb')

File [~/Documents/spyglass/src/spyglass/data_import/insert_sessions.py:68](http://localhost:8892/lab/tree/Documents/spyglass/notebooks/DataPreprocessing/Documents/spyglass/src/spyglass/data_import/insert_sessions.py#line=67), in insert_sessions(nwb_file_names)
     64 # Make a copy of the NWB file that ends with '_'.
     65 # This has everything except the raw data but has a link to
     66 # the raw data in the original file
     67 copy_nwb_link_raw_ephys(nwb_file_name, out_nwb_file_name)
---> 68 Nwbfile().insert_from_relative_file_name(out_nwb_file_name)
     69 populate_all_common(out_nwb_file_name)

File [~/Documents/spyglass/src/spyglass/common/common_nwbfile.py:72](http://localhost:8892/lab/tree/Documents/spyglass/notebooks/DataPreprocessing/Documents/spyglass/src/spyglass/common/common_nwbfile.py#line=71), in Nwbfile.insert_from_relative_file_name(cls, nwb_file_name)
     70 key["nwb_file_name"] = nwb_file_name
     71 key["nwb_file_abs_path"] = nwb_file_abs_path
---> 72 cls.insert1(key, skip_duplicates=True)

File [~/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:337](http://localhost:8892/lab/tree/Documents/spyglass/notebooks/DataPreprocessing/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py#line=336), in Table.insert1(self, row, **kwargs)
    330 def insert1(self, row, **kwargs):
    331     """
    332     Insert one data record into the table. For ``kwargs``, see ``insert()``.
    333 
    334     :param row: a numpy record, a dict-like object, or an ordered sequence to be inserted
    335         as one row.
    336     """
--> 337     self.insert((row,), **kwargs)

File [~/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:419](http://localhost:8892/lab/tree/Documents/spyglass/notebooks/DataPreprocessing/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py#line=418), in Table.insert(self, rows, replace, skip_duplicates, ignore_extra_fields, allow_direct_insert)
    416     return
    418 field_list = []  # collects the field list from first row (passed by reference)
--> 419 rows = list(
    420     self.__make_row_to_insert(row, field_list, ignore_extra_fields)
    421     for row in rows
    422 )
    423 if rows:
    424     try:

File [~/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:420](http://localhost:8892/lab/tree/Documents/spyglass/notebooks/DataPreprocessing/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py#line=419), in <genexpr>(.0)
    416     return
    418 field_list = []  # collects the field list from first row (passed by reference)
    419 rows = list(
--> 420     self.__make_row_to_insert(row, field_list, ignore_extra_fields)
    421     for row in rows
    422 )
    423 if rows:
    424     try:

File [~/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:872](http://localhost:8892/lab/tree/Documents/spyglass/notebooks/DataPreprocessing/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py#line=871), in Table.__make_row_to_insert(self, row, field_list, ignore_extra_fields)
    870 elif isinstance(row, collections.abc.Mapping):  # dict-based
    871     check_fields(row)
--> 872     attributes = [
    873         self.__make_placeholder(name, row[name], ignore_extra_fields)
    874         for name in self.heading
    875         if name in row
    876     ]
    877 else:  # positional
    878     try:

File [~/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:873](http://localhost:8892/lab/tree/Documents/spyglass/notebooks/DataPreprocessing/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py#line=872), in <listcomp>(.0)
    870 elif isinstance(row, collections.abc.Mapping):  # dict-based
    871     check_fields(row)
    872     attributes = [
--> 873         self.__make_placeholder(name, row[name], ignore_extra_fields)
    874         for name in self.heading
    875         if name in row
    876     ]
    877 else:  # positional
    878     try:

File [~/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:832](http://localhost:8892/lab/tree/Documents/spyglass/notebooks/DataPreprocessing/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py#line=831), in Table.__make_placeholder(self, name, value, ignore_extra_fields)
    826         value = (
    827             str.encode(attachment_path.name)
    828             + b"\0"
    829             + attachment_path.read_bytes()
    830         )
    831 elif attr.is_filepath:
--> 832     value = self.external[attr.store].upload_filepath(value).bytes
    833 elif attr.numeric:
    834     value = str(int(value) if isinstance(value, bool) else value)

File [~/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/external.py:279](http://localhost:8892/lab/tree/Documents/spyglass/notebooks/DataPreprocessing/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/external.py#line=278), in ExternalTable.upload_filepath(self, local_filepath)
    276 if check_hash:
    277     # the tracking entry exists, check that it's the same file as before
    278     if contents_hash != check_hash[0]:
--> 279         raise DataJointError(
    280             f"A different version of '{relative_filepath}' has already been placed."
    281         )
    282 else:
    283     # upload the file and create its tracking entry
    284     self._upload_file(
    285         local_filepath,
    286         self._make_external_filepath(relative_filepath),
    287         metadata={"contents_hash": str(contents_hash)},
    288     )

DataJointError: A different version of 'lewis20240115_.nwb' has already been placed.

But data had been deleted through (nd.common.Nwbfile() & {"nwb_file_name": "lewis20240115.nwb"}).delete()

Expected behavior Calling delete before insert should allow me to reinsert data.

CBroz1 commented 1 month ago

Hi @shijiegu - Looking at your error stack versus the latest commit (e.g., insert_sessions.py:L68 vs master), it appears you're on an older version. I believe we've addressed this issue. Please feel free to reopen if this is still a problem after fetching the latest updates.

In older versions of Spyglass, I believe files need to be deleted from disk manually.