Dandi export of nwb w/ external files

[samuelbray32]

Describe the bug

• Dandi has specific rules about nwb with external files.
  • Key example: raw nwb file with video
• Our current organizing of the dandiset here results in the dandiset skipping all nwb with external files
• dandi.organize.orgaize does provide an argument update_external_filepaths that fixes the issue to dandi standards and allows inclusion
  • However, this does edit the nwb file in the paper_dir/destination_dir folder
  • This currently is a symlink to the file on server. Edits would edit that file and break checksum

Proposed Solution

• when putting symlinks in the paper_dir/destination_dir folder, check if the nwb has eternal filepaths
  • If so, copy the file to the folder instead
  • This is memory inneficient, but the file is only needed temporarily so can delete after
  • Open to other ideas here as well!
• Now can allow update_external_filepaths=True in dandi.orgaize call.
  • edits the copied files with external filepaths, leaves originals inact
  • No checksum issue created or changes to originals
• Files with external filepaths uploaded to dandi will have different checksum than database
  • fetch_nwb from dandi source doesn't trigger a checksum check so shouldn't be an issue

[samuelbray32]

Update

• Dandi will not accept the copied_.nwb files because they contain external links to objects in the original nwb file (e.g. acquisition['e-series']
• We will instead need to upload the original nwb files to DANDI

Issues

• Since the original nwb file is not tracked in spyglass the export process will need to find it and get it into the dandiset
• Edits to external filepaths will affect the original unless we copy the whole file (very expensive)
• Will need a tool to recreate the copy_.nwb file within the docker image filesystem from the original uploaded to DANDI