search

catalystneuro / neuroconv

Create NWB files by converting and combining neural data in proprietary formats and adding essential metadata.
https://neuroconv.readthedocs.io
BSD 3-Clause "New" or "Revised" License
51 stars 22 forks source link

[Bug]: Miniscope creates absolute paths #1105

Open bendichter opened 1 month ago

bendichter commented 1 month ago

What happened?

from Laurel Keyes on the NWB Slack:

Hello all, I have a best practice violation as follows:

check_image_series_external_file_relative - 'ImageSeries' object at location '/acquisition/BehavCamImageSeries'
      Message: The external file '<ABSOLUTE_PATH_TO_FILE>/<SUBJECT_ID>/18_38_48/BehavCam_2/0.avi' is not a relative path. Please adjust the absolute path to be relative to the location of the NWBFile.

I am using the neuroconv converter for Miniscopes and it seems to be automatically creating the links to the behavioral video snippets that accompany each neural recording. Is there an easy way to make these paths relative or do I need to update each one manually (I have about 30 per nwb file)? Or, am I able to ignore them, since I ultimately plan to upload to dandiarchive and I will update the external file paths when I do. Thanks for the guidance!

Steps to Reproduce

n.a.

Traceback

No response

Operating System

Windows

Python Executable

Conda

Python Version

3.8

Package Versions

No response

Code of Conduct

bendichter commented 1 month ago

I think the fix here would be to add nwbfile_path as an arg of add_to_nwbfile and use the relative path of each external file wrt that path.

laurelrr commented 1 month ago

If it helps, I am working from the code base here: https://github.com/catalystneuro/tye-lab-to-nwb/tree/main/src/tye_lab_to_nwb/ast_ophys

weiglszonja commented 1 month ago

hey! I think you can ignore this, since you'll upload this files to DANDI you can use dandi organize to update the file paths for you.

Assuming you have the NWB files that you want to upload in folder with the dandiset.yaml file:

/ ├── file1.nwb ├── file2.nwb ├── ... ├── file30.nwb └── dandiset.yaml From the terminal you can do: ``` cd dandi organize . --update-external-file-paths --files-mode=move --media-files-mode=copy ```
laurelrr commented 1 month ago

OK, I did not get an error message when running this command. 

$ dandi organize ../ --update-external-file-paths --files-mode=move --media-files-mode=copy
2024-10-02 13:03:44,979 [    INFO] Note: detected 128 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
2024-10-02 13:03:44,980 [    INFO] Note: NumExpr detected 128 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2024-10-02 13:03:44,980 [    INFO] NumExpr defaulting to 8 threads.
2024-10-02 13:03:46,657 [    INFO] Loading metadata from 7 files
[Parallel(n_jobs=-1)]: Using backend LokyBackend with 128 concurrent workers.
[Parallel(n_jobs=-1)]: Done   1 tasks      | elapsed:    4.0s
[Parallel(n_jobs=-1)]: Done   2 out of   7 | elapsed:    4.0s remaining:   10.0s
[Parallel(n_jobs=-1)]: Done   3 out of   7 | elapsed:    4.0s remaining:    5.4s
[Parallel(n_jobs=-1)]: Done   4 out of   7 | elapsed:    4.0s remaining:    3.0s
[Parallel(n_jobs=-1)]: Done   5 out of   7 | elapsed:    4.1s remaining:    1.6s
[Parallel(n_jobs=-1)]: Done   7 out of   7 | elapsed:    4.1s finished
2024-10-02 13:08:15,317 [    INFO] Organized 7 paths. Visit /snlkt/ast/ASt_NWB/NWB_Masters/calcium_ophys/nwbfiles/001203/
2024-10-02 13:08:15,320 [    INFO] Logs saved in /home/lkeyes/.local/state/dandi-cli/log/2024.10.02-20.03.43Z-3205073.log

And it looks like it created the 7 subject folders in my folder. Each subject folder has the nwb file and a folder of videos. They are labeled in a non-intuitive manner but that's ok:

$ ll sub-J650/sub-J650_ses-C6-J650-Disc5_image+ophys/
total 1881528
drwxrwsr-x 2 lkeyes snlkt     4096 Oct  2 13:05 ./
drwxrwsr-x 3 lkeyes snlkt     4096 Oct  2 13:05 ../
-rw-rw-r-- 1 lkeyes snlkt 64735438 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_0.avi
-rw-rw-r-- 1 lkeyes snlkt 64017402 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_10.avi
-rw-rw-r-- 1 lkeyes snlkt 63957728 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_11.avi
-rw-rw-r-- 1 lkeyes snlkt 64042354 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_12.avi
-rw-rw-r-- 1 lkeyes snlkt 64025076 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_13.avi
-rw-rw-r-- 1 lkeyes snlkt 63872320 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_14.avi
-rw-rw-r-- 1 lkeyes snlkt 63973374 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_15.avi
-rw-rw-r-- 1 lkeyes snlkt 63834236 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_16.avi
-rw-rw-r-- 1 lkeyes snlkt 63774558 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_17.avi
-rw-rw-r-- 1 lkeyes snlkt 63784944 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_18.avi
-rw-rw-r-- 1 lkeyes snlkt 63790340 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_19.avi
-rw-rw-r-- 1 lkeyes snlkt 64020430 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_1.avi
-rw-rw-r-- 1 lkeyes snlkt 63851704 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_20.avi
-rw-rw-r-- 1 lkeyes snlkt 63689290 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_21.avi
-rw-rw-r-- 1 lkeyes snlkt 63636148 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_22.avi
-rw-rw-r-- 1 lkeyes snlkt 63866866 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_23.avi
-rw-rw-r-- 1 lkeyes snlkt 63902258 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_24.avi
-rw-rw-r-- 1 lkeyes snlkt 63942888 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_25.avi
-rw-rw-r-- 1 lkeyes snlkt 63728326 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_26.avi
-rw-rw-r-- 1 lkeyes snlkt 63760076 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_27.avi
-rw-rw-r-- 1 lkeyes snlkt 63868856 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_28.avi
-rw-rw-r-- 1 lkeyes snlkt 63741112 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_29.avi
-rw-rw-r-- 1 lkeyes snlkt 64085406 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_2.avi
-rw-rw-r-- 1 lkeyes snlkt 64170392 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_3.avi
-rw-rw-r-- 1 lkeyes snlkt 64494640 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_4.avi
-rw-rw-r-- 1 lkeyes snlkt 64121422 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_5.avi
-rw-rw-r-- 1 lkeyes snlkt 64175404 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_6.avi
-rw-rw-r-- 1 lkeyes snlkt 64033816 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_7.avi
-rw-rw-r-- 1 lkeyes snlkt 63996208 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_8.avi
-rw-rw-r-- 1 lkeyes snlkt 63992862 Oct  2 13:05 4a92f34b-c0f9-4124-8184-80a2a1b258f1_external_file_9.avi

Thanks!