Open jgoodson opened 1 year ago
I will add documentation about absolute paths in README.
Caper's localization engine autouri
cannot distinguish between relative path and plain string, so was not able to add it to Singularity's bindpath. Currently, it's recommended to use absolute paths only in an input JSON particularly for Singularity.
Thanks for reporting, I will look into this and fix it soon. Please use absolute paths until it's fixed.
This is affecting our environment as well. Since cromwell localization by default does hard link, soft link and file copy with cromwell. Most institutes have labs and multiple filesystems that shares data and in this case, singularity container seems to be unable to access data that are localized by cromwell and turned into relative paths.
We faced this issue as well, and using absolute paths did not work for us. To solve this, I created a custom backend and changed the order of the localization list to:
localization = [
"hard-link"
"copy"
"soft-link"
]
and then passed the custom backend file using --backend-file
via cmd line.
@sidwekhande Are you using slurm
backend? It's weird that changing the order fixed the problem. I think you can fix it by not including linked (symlinked or hardlinked) files in input JSON.
For example, if you have an original file at /home/me/original/genome.tsv
and it's symlinked to /home/me/linked/genome.tsv
. You should not use /home/me/linked/genome.tsv
. Simply using original paths in input JSON will fix the problem.
Make sure that all files defined in genome.tsv
are not linked (soft or hard) either. e.g. fasta, genome size files, bowtie2 indices.
I will add this to my ToDo list. I will need to edit caper.singularity.find_bindpath()
function to recognize linked files and find original ones (recursively for files in genome TSV too).
{
"atac.genome_tsv": "/home/me/original/genome.tsv"
}
Please let me know if u can run it without adding your own --backend-file
.
This error occurs while using the atac-seq-pipeline but I do not believe this is specific to that. When using Singularity, relative paths to local input files do not get included in the Singularity bindpath. This causes jobs to fail when the input files are symlinked back to the original location (the default first-priority for localization) as the base directory is not bound into the container. Changing the paths to absolute paths fixes this issue.
This error will look something like:
In this example, that file is a symlink to a file in
/gpfs/gsfs8/users/goodsonjr/encode-atac/input/
. The submitted script invokes singularity with this command:This generated bindpaths for the
atac.genome_tsv
file as it was an absolute path, but the relative paths to the FastQ files are discarded. Since the original files aren't in--bind
or the--home
path, the container cannot read the file.Details: Looking at the Caper code, it runs
caper.singularity.find_bindpath()
on the input JSON file to determine what paths to bind-mount. This function callsautouri.AbsPath
to determine if it is a valid path. It then uses some logic to determine which parent directories to bind-mount. Since this function takes the relative paths and directly generates an autouriAbsPath
, the relative paths don't generate valid URIs, and won't be included in the bind-path generation logic. I get a comparable result when calling this function directly, this conditional:https://github.com/ENCODE-DCC/caper/blob/2ff0999285490984e43fa8d0f74f636575f973c8/caper/singularity.py#L43-L45
evaluates to
False
when fed a relative path. This means the path won't be included inall_dirnames
and won't contribute to the bindpath.This issue does not seem to arise with the plain local backend without Slurm. I haven't figured out why, but when using Slurm Caper creates symlinks in the workflow run directory, while with the local backend they get copied or hardlinked, despite the generated backend.conf having the same order for
backend.providers.Local.config.filesystem.local.localization
.I looked but was unable to find any documentation concerning absolute vs relative paths, and the descriptions of the input JSON format use either web URIs or relative local paths. I am not sure what to suggest, although using
os.path.abspath()
to convert relative paths to absolute before generating theautouri.AbsPath
might potentially resolve this.