search

shuzhao-li-lab / PythonCentricPipelineForMetabolomics

Python pipeline for metabolomics data preprocessing, QC, standardization and annotation
Other
8 stars 0 forks source link

Cross-device link" error when running assemble #66

Open gmhhope opened 6 months ago

gmhhope commented 6 months ago

Cross-device link" error when running assemble

Describe the bug When running the pcpfm tool, an OSError with code 18 occurs, indicating a "Cross-device link" error. This error arises when attempting to create a hard link between files located on different devices.

pcpfm assemble -s  serum_Rp.csv --name_field='rfID' --path_field='File_path' -o ../../output/pfmRun0415/ -j serum_Rp

[Warning] Not index found and build_index_from_scratch is False
[Warning] Not index found and build_index_from_scratch is False
Traceback (most recent call last):
  File "/Users/gongm/mambaforge/bin/pcpfm", line 8, in <module>
    sys.exit(CLI())
  File "/Users/gongm/mambaforge/lib/python3.10/site-packages/pcpfm/main.py", line 875, in CLI
    main()
  File "/Users/gongm/mambaforge/lib/python3.10/site-packages/pcpfm/main.py", line 863, in main
    function(params)
  File "/Users/gongm/mambaforge/lib/python3.10/site-packages/pcpfm/main.py", line 276, in assemble
    experiment = Experiment.Experiment.construct_experiment_from_CSV(
  File "/Users/gongm/mambaforge/lib/python3.10/site-packages/pcpfm/Experiment.py", line 589, in construct_experiment_from_CSV
    experiment.add_acquisition(acquisition)
  File "/Users/gongm/mambaforge/lib/python3.10/site-packages/pcpfm/Experiment.py", line 403, in add_acquisition
    file_operations[mode](acquisition.source_filepath, target_path)
OSError: [Errno 18] Cross-device link: '/Volumes/LM_MF_LaCie/kidney-acidosis/Proj-117-Oh-Korstanje-Li-microbial-kidney-acidosis/mzML_rn/serum/serum_Rp/SolventBlank_20230907_002_sr.mt_bc01.ij2.Rp.mzML' -> '/Users/gongm/Documents/projects/kidney-acidosis_project/git-repo/output/pfmRun0415/serum_Rp/converted_acquisitions/SolventBlank_20230907_002_sr.mt_bc01.ij2.Rp.mzML'

To Reproduce Steps to reproduce the behavior:

  1. Execute the pcpfm command with specific parameters.
  2. Put mzML files in external hard drive
  3. Then run pcpfm assemble script.
  4. Output could be external hard drive or in macbook pro (both seems to have errors)

Expected behavior The pcpfm tool should successfully add the acquisition without encountering a cross-device link error.

Screenshots N/A

Desktop

Additional context This error likely arises due to attempting to copy or move files across different filesystems or partitions. The pcpfm tool could be modified to handle copying files across devices more gracefully, perhaps by using a method that automatically handles such scenarios

(generated from chatGPT)

jmmitc06 commented 5 months ago

This is largely OS dependent. Most do not allow symbolic links across device partitions. I have no intention of fixing this directly because that is probably a restriction for a good reason, rather, you can use '--file_mode=copy' in the assemble step to copy the files instead of symbolically linking them. This will however, make a copy of the file to the partition where the pcpfm is being ran from which may or may not be a good thing depending on your use case. Alternatively if you have your files on an external drive, cd to the drive partition and process the data there.

jmmitc06 commented 2 months ago

Just FYI. This requirement was only due to Asari. Now Asari no longer needs all the files to be in the same place (will be merged into V2). In a future version of the pipeline, this will be fixed as files will no longer need to be moved/linked/copied at all during processing.

I'm going to reopen the issue as now I intend to fix this as it is fixed in Asari too.