Calls in the export process that don't trigger logging

[samuelbray32]

Describe the bug In testing the export and docker pipeline, found several cases where entries weren't logged for the SQL dump that we should probably include or make clear in documentation wont be hit. Listing here:

Using a table to restrict another

• There were cases in my code where I select entries in table1 via table1 & (table2 & key) or table1 * (table2 & key)
• Since i never fetch from table2 it doesn't get exported. This then changes behavior of the restriction in the docker
• This could also be solved on the user end by ensuring call made as table1 & (table2 & key).fetch("KEY")

__fetch_nwb__

• calling fetch_nwb from the merge table did not register the merge table entries in the export.
• The entries of the parent source table were registered
• Call in the code tracked for export: (LFPOutput & basic_key).fetch_nwb(restriction=basic_key)[0]["lfp"] where basic_key has restrictions for the source table LFPV1()

[samuelbray32]

__fetch_nwb notes:__

For the Merge.fetch_nwb() case:

• Merge table entry: Not included
• Parent Source entry: included
• AnalysisNwbfile entry: not included
• DandiPath: included

Tracing:

Missing Merge entry

• Merge.fetch_nwb() doesn't do any logging before calling ParentSource.fetch_nwb(). explains why this not included
• Maybe add in export logging within Merge.merge_restrict_class() to catch cases where the merge table is a "passthrough" for upstream tables

Missing AnalysisNwbfile entry

• _log_fetch in the mixin ignores fetch calls from _get_abspath here
• This means it's not logged when get_nwb_table calls get_abs_path here
• fetch_nwb takes the filepath it gets from here and loads it with get_nwb_file. If this is an unopened local file, it accesses it with a pynwb io function that doesn't require any more fetch calls
  • This entry is therefore never logged for export in the AnalysisNwbfile table
  • However, SpyglassMixin.fetch_nwb() logs the file for dandi export and the table entry of the ParentSource

It seems like the AnalysisNwbfile is maybe not getting hit as expected in the restriction graph from the SourceTable entry? (@CBroz1 )

[CBroz1]

Thanks for putting this together @samuelbray32. I'll see what I can do about either capturing these missed cases or flagging them as items that the process would miss

[CBroz1]

Fixed in #1164