preprocess.json example and help with errors on HPC

Hello,

I am trying to run the SHARE-Seq pipeline starting from the preprocessing step on an HPC.

Could someone share an example JSON and other required preprocessing files for starting the analysis at the preprocessing stage?

I created a JSON file following the subwf_preprocess.json example, but encountered an error due to missing files:

wf_preprocess.metaCsv wf_preprocess.bcl wf_preprocess.workspace_name wf_preprocess.terra_project

I tried specifying these as null and provided a workspace_name ("test1"), but it didn't resolve the issue. I also added other sections that were mentioned as required in the input JSON documentation, but still without success.

A different error was "Singularity image not found in WDL metadata." I am running Docker through Singularity, should I modify the share-seq.wdl to account for this, or is there another solution?

Any help would be appreciated! Nayeli

Errors:

$ ml singularity/3.6.4

$ INPUT_JSON="/nayeli/single-cell-training/share-seq/epi-SHARE-seq-pipeline/test1/json/subwf_preprocess.json" 

$ caper run subwf-preprocess.wdl -i "${INPUT_JSON}" --singularity --max-concurrent-tasks 1
2024-10-11 16:01:25,162|caper.cli|INFO| Cromwell stdout: /nayeli/software/epi_share_seq_pipeline/workflows/cromwell.out
2024-10-11 16:01:25,171|caper.caper_base|INFO| Creating a timestamped temporary directory. /nayeli/software/caper/subwf-preprocess/20241011_160125_167774
2024-10-11 16:01:25,171|caper.caper_runner|INFO| Localizing files on work_dir. /nayeli/software/caper/subwf-preprocess/20241011_160125_167774
2024-10-11 16:01:25,445|caper.caper_workflow_opts|INFO| Singularity image not found in WDL metadata. wdl=/nayeli/software/epi_share_seq_pipeline/workflows/subwf-preprocess.wdl.
2024-10-11 16:01:25,462|caper.cromwell|INFO| Validating WDL/inputs/imports with Womtool...
2024-10-11 16:01:27,572|caper.nb_subproc_thread|ERROR| Subprocess failed. returncode=1
Traceback (most recent call last):
  File "/sc/arion/scratch/gutien07/test-env/envs/rocky9_radian/bin/caper", line 13, in <module>
    main()
  File "/sc/arion/scratch/gutien07/test-env/envs/rocky9_radian/lib/python3.12/site-packages/caper/cli.py", line 710, in main
    return runner(parsed_args, nonblocking_server=nonblocking_server)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sc/arion/scratch/gutien07/test-env/envs/rocky9_radian/lib/python3.12/site-packages/caper/cli.py", line 255, in runner
    subcmd_run(c, args)
  File "/sc/arion/scratch/gutien07/test-env/envs/rocky9_radian/lib/python3.12/site-packages/caper/cli.py", line 385, in subcmd_run
    thread = caper_runner.run(
             ^^^^^^^^^^^^^^^^^
  File "/sc/arion/scratch/gutien07/test-env/envs/rocky9_radian/lib/python3.12/site-packages/caper/caper_runner.py", line 462, in run
    self._cromwell.validate(wdl=wdl, inputs=inputs, imports=imports)
  File "/sc/arion/scratch/gutien07/test-env/envs/rocky9_radian/lib/python3.12/site-packages/caper/cromwell.py", line 159, in validate
    raise WomtoolValidationFailed(
caper.cromwell.WomtoolValidationFailed: RC=1
STDERR=Required workflow input 'wf_preprocess.metaCsv' not specified
Required workflow input 'wf_preprocess.bcl' not specified
Required workflow input 'wf_preprocess.workspace_name' not specified
Required workflow input 'wf_preprocess.terra_project' not specified

HPC environment:

ml

Currently Loaded Modules:
  1) singularity/3.6.4

 (base) li03c04:/sc/arion/scratch/gutien07/test-env/envs/rocky9_radian/lib/python3.12/site-packages/caper
$ which singularity
/hpc/packages/minerva-centos7/singularity/3.6.4/bin/singularity
(base) li03c04:/sc/arion/scratch/gutien07/test-env/envs/rocky9_radian/lib/python3.12/site-packages/caper
$ singularity --version
singularity version 3.6.4
(base) li03c04:/sc/arion/scratch/gutien07/test-env/envs/rocky9_radian/lib/python3.12/site-packages/caper
$ singularity exec docker://ubuntu:latest echo "Singularity and Docker are integrated"
INFO:    Using cached SIF image
Singularity and Docker are integrated
(base) li03c04:/sc/arion/scratch/gutien07/test-env/envs/rocky9_radian/lib/python3.12/site-packages/caper
$ radian --version

radian version: 0.6.13
r executable: /sc/arion/scratch/gutien07/test-env/envs/rocky9_radian/lib/R/bin/R
r version: 4.2.0
python executable: /sc/arion/scratch/gutien07/test-env/envs/rocky9_radian/bin/python3.12
python version: 3.12.6
(base) li03c04:/sc/arion/scratch/gutien07/test-env/envs/rocky9_radian/lib/python3.12/site-packages/caper
$ R --version
R version 4.2.0 (2022-04-22) -- "Vigorous Calisthenics"
Copyright (C) 2022 The R Foundation for Statistical Computing
Platform: x86_64-conda-linux-gnu (64-bit)

broadinstitute / epi-SHARE-seq-pipeline

preprocess.json example and help with errors on HPC #161