search

LorenFrankLab / spyglass

Neuroscience data analysis framework for reproducible research built by Loren Frank Lab at UCSF
https://lorenfranklab.github.io/spyglass/
MIT License
94 stars 43 forks source link

populate error due to upstream table change #1115

Closed rpswenson closed 2 months ago

rpswenson commented 2 months ago

Running this populate for SpikeSorting:

key = {'recording_id': 'ddaced8f-87fc-4e04-80d1-f430e4db7f2d', 'sorter': 'mountainsort4', 'sorter_param_name': 'franklab_probe_ctx_30KHz', 
       'nwb_file_name': 'RS1220240809_.nwb', 'interval_list_name': '1eea1a3b-bd65-450c-a377-52b1ab4ee349'}

sss_pk = (sgs.SpikeSortingSelection & key).proj()
sgs.SpikeSorting.populate(sss_pk)

I get this output:

[13:11:23][WARNING] Spyglass: Turning off transaction protection this table by default. Use use_transation=True to re-enable.
Read more about transactions:
https://docs.datajoint.io/python/definition/05-Transactions.html
https://github.com/LorenFrankLab/spyglass/issues/1030
[2024-09-16 13:11:26,774][WARNING]: Skipped checksum for file with hash: 71bda2b1-ce6b-e240-e9d3-bc1a8b4dad06, and path: /stelmo/nwb/raw/RS1220240809_.nwb
[2024-09-16 13:11:26,784][WARNING]: Skipped checksum for file with hash: 3acc0e03-4803-9021-d43a-0ecbc0225379, and path: /stelmo/nwb/analysis/RS1220240809/RS1220240809_TTNX3Q05A6.nwb
[2024-09-16 13:11:26,854][WARNING]: Skipped checksum for file with hash: 3acc0e03-4803-9021-d43a-0ecbc0225379, and path: /stelmo/nwb/analysis/RS1220240809/RS1220240809_TTNX3Q05A6.nwb

Mountainsort4 use the OLD spikeextractors mapped with NewToOldRecording

[13:17:45][INFO] Spyglass: Writing new NWB file RS1220240809_T0SWORQ8BR.nwb
/home/rswenson/miniforge3/envs/spyglass/lib/python3.9/site-packages/datajoint/hash.py:39: ResourceWarning: unclosed file <_io.BufferedReader name='/stelmo/nwb/analysis/RS1220240809/RS1220240809_T0SWORQ8BR.nwb'>
  return uuid_from_stream(Path(filepath).open("rb"), init_string=init_string)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/home/rswenson/miniforge3/envs/spyglass/lib/python3.9/site-packages/datajoint/external.py:276: DeprecationWarning: The truth value of an empty array is ambiguous. Returning False, but in future this will result in an error. Use `array.size > 0` to check that an array is not empty.
  if check_hash:
/home/rswenson/miniforge3/envs/spyglass/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/stelmo/nwb/tmp/tmpdcd4yiim'>
  _warnings.warn(warn_message, ResourceWarning)
[2024-09-16 13:17:53,506][WARNING]: Skipped checksum for file with hash: 71bda2b1-ce6b-e240-e9d3-bc1a8b4dad06, and path: /stelmo/nwb/raw/RS1220240809_.nwb
[2024-09-16 13:17:53,511][WARNING]: Skipped checksum for file with hash: 3acc0e03-4803-9021-d43a-0ecbc0225379, and path: /stelmo/nwb/analysis/RS1220240809/RS1220240809_TTNX3Q05A6.nwb
[13:17:53][INFO] Spyglass: Path Found : ['`common_session`.`_session`', '`common_interval`.`interval_list`'...
[13:17:53][INFO] Spyglass: Checking  0: SpikeSorting -> ['SpikeSortingSelection', 'AnalysisNwbfile']
[13:17:53][INFO] Spyglass: Bridge Link: SpikeSorting -> SpikeSortingSelection: result partial
[13:17:53][INFO] Spyglass: Checking  1: SpikeSortingSelection -> ['SpikeSortingRecording', 'SpikeSorterPara...
[13:17:53][INFO] Spyglass: Banned Tbl : SpikeSortingRecording
[13:17:53][INFO] Spyglass: Banned Tbl : SpikeSorterParameters
[13:17:53][INFO] Spyglass: Banned Tbl : IntervalList
[13:17:53][INFO] Spyglass: Banned Tbl : AnalysisNwbfile
[13:17:53][INFO] Spyglass: No restr for Session
[13:17:53][INFO] Spyglass: Queueing delete for session(s):
*nwb_file_name *lab_member_na
+------------+ +------------+

 (Total: 0)

[2024-09-16 13:17:53,681][INFO]: Deleting 1 rows from `spikesorting_v1_sorting`.`__spike_sorting`
[2024-09-16 13:17:57,793][INFO]: Deleting 0 rows from `common_interval`.`interval_list`
/home/rswenson/miniforge3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:644: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
  logger.warn("Nothing to delete.")
[2024-09-16 13:17:57,795][WARNING]: Nothing to delete.
[13:17:57][ERROR] Spyglass: Upstream tables changed during non-transaction populate. Please try again.

This particular output also has a "nothing to delete" warning but when I first ran into this in my spikesorting loop it did not get that warning, but still had the Upstream tables changed during non-transaction populate. Please try again. message

CBroz1 commented 2 months ago

Nothing to delete is from the delete call to IntervalList, so we can ignore that. If you find it distracting, we can revisit the delete logic to first check for deletes before running. That would trade off speed for output clarity

Upstream tables error is our way of protecting data integrity without transactions. More info here. I adjusted the check mechanism in #1108 yesterday. Updating may fix the issue. It may be the case that someone else was working on something upstream at the same time, in which case this is working as intended. If you encounter this issue repeatedly, it could be a bug.

This may work better:

sss_pk = (sgs.SpikeSortingSelection & key).proj().fetch(as_dict=True)
sgs.SpikeSorting.populate(sss_pk)

Please use the bug report template for future bug submissions

rpswenson commented 2 months ago

okay yeah updating fixed it. apologies