Run error (singularity)

[Artifice120]

Afternoon,

While running the following command on a genome assembly ;

python3 ui/egapx.py ./examples/input_Ue_polished.yaml -e singularity -w tmp/ -o Ue_out

I received the following error;

Command error:                                                                                                                                                                                                     INFO:    /etc/singularity/ exists; cleanup by system administrator is not complete (see https://apptainer.org/docs/admin/latest/singularity_migration.html)                                                      INFO:    Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred                                                                                                                 INFO:    Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred                                                                                             INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred                                                                                                           split: invalid number of chunks: '0': Numerical result out of range                                                                                                                                                                                                                                                                                                                                                             Work dir:                                                                                                                                                                                                          /lustre/isaac/scratch/jtorre28/unstable/egapx/tmp/82/74c865bc18108ebe530bf9cf6782f3                                                                                                                                                                                                                                                                                                                                             Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`                                                                                                                                                                                                                                                                                                              -- Check '/lustre/isaac/scratch/jtorre28/unstable/egapx/Ue_out/nextflow.log' file for details                                                                                                                   WARN: Killing running tasks (3)                                                                                                                                                                                                                                                                                                                                                                                                   executor >  local (11)                                                                                                                                                                                           [-        ] ega…tup_genome:get_genome_info -
[7e/2e4757] egapx:run_get_hmm              | 1 of 1 ✔
[3a/1d2f52] ega…ion:chainer:run_align_sort | 1 of 1 ✔
[-        ] ega…tion:chainer:generate_jobs | 0 of 1
[-        ] ega…ration:chainer:run_chainer -
[-        ] ega…ainer:run_gpx_make_outputs -
[-        ] ega…n:gnomon_wnode:gpx_qsubmit -
[-        ] ega…eration:gnomon_wnode:annot -
[-        ] ega…ion:gnomon_wnode:gpx_qdump -
[-        ] ega…aining:run_gnomon_training -
[15/693196] ega…on2:chainer:run_align_sort | 1 of 1 ✔
[82/74c865] ega…ion2:chainer:generate_jobs | 1 of 1, failed: 1 ✘
[-        ] ega…ation2:chainer:run_chainer -
[-        ] ega…ainer:run_gpx_make_outputs -
[-        ] ega…2:gnomon_wnode:gpx_qsubmit -
[-        ] ega…ration2:gnomon_wnode:annot -
[-        ] ega…on2:gnomon_wnode:gpx_qdump -
[-        ] ega…aining:run_gnomon_training -
[ee/e68266] ega…on3:chainer:run_align_sort | 1 of 1 ✔
[-        ] ega…ion3:chainer:generate_jobs | 0 of 1
[81/6d27cf] ega…on4:chainer:run_align_sort | 1 of 1 ✔
[63/9e3423] egapx:chainer:run_align_sort   | 1 of 1 ✔
[69/8b81b4] ega…builder:annot_builder_main | 1 of 1 ✔                                                                                                                                                            Plus 24 more processes waiting for tasks…                                                                                                                                                                        Pulling Singularity image docker://ncbi/egapx:latest [cache /lustre/isaac/scratch/jtorre28/unstable/egapx/tmp/singularity/ncbi-egapx-latest.img]                                                                 WARN: Singularity cache directory has not been defined -- Remote image will be stored in the path: /lustre/isaac/scratch/jtorre28/unstable/egapx/tmp/singularity -- Use the environment variable NXF_SINGULARITY_CACHEDIR to specify a different location
ERROR ~ Error executing process > 'egapx:gnomon_training_iterations:gnomon_training_iteration2:chainer:generate_jobs'

Caused by:
  Missing output file(s) `job.*` expected by process `egapx:gnomon_training_iterations:gnomon_training_iteration2:chainer:generate_jobs`
Command executed:

  #!/usr/bin/env bash
  # generate_jobs sorted_aligns.asn -minimum-abut-margin 20 -separate-within-introns -output chains -output-slices chains_slices -output-evidence evidence -output-evidence-slices evidence_slices
  submit_chainer -minimum-abut-margin 20 -separate-within-introns -asn sorted_aligns.asn -o jobs
  total_lines=$(wc -l <jobs)
  (( lines_per_file = (total_lines + 16 - 1) / 16 ))
  echo total_lines=$total_lines, lines_per_file=$lines_per_file
  ####split -l$lines_per_file jobs job. -da 3
  # Use round robin to distribute jobs across nodes more evenly
  if [ $total_lines -lt 16 ]; then
      effective_njobs=$total_lines
  else
      effective_njobs=16
  fi
  split -nr/$effective_njobs jobs job. -da 3

Command exit status:
  0

Command output:
  total_lines=0, lines_per_file=0

Command error:
  INFO:    /etc/singularity/ exists; cleanup by system administrator is not complete (see https://apptainer.org/docs/admin/latest/singularity_migration.html)
  INFO:    Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
  split: invalid number of chunks: '0': Numerical result out of range

Work dir:
  /lustre/isaac/scratch/jtorre28/unstable/egapx/tmp/82/74c865bc18108ebe530bf9cf6782f3

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

 -- Check '/lustre/isaac/scratch/jtorre28/unstable/egapx/Ue_out/nextflow.log' file for details

!!WARNING!!
This is an alpha release with limited features and organism scope to collect initial feedback on execution. Outputs are not yet complete and not intended for production use.

None
To resume execution, run:
nextflow -C /lustre/isaac/scratch/jtorre28/unstable/egapx/egapx_config/singularity.config,/lustre/isaac/scratch/jtorre28/unstable/egapx/ui/assets/config/default.config,/lustre/isaac/scratch/jtorre28/unstable/egapx/ui/assets/config/docker_image.config,/lustre/isaac/scratch/jtorre28/unstable/egapx/ui/assets/config/process_resources.config -log /lustre/isaac/scratch/jtorre28/unstable/egapx/Ue_out/nextflow.log run /lustre/isaac/scratch/jtorre28/unstable/egapx/ui/../nf/ui.nf --output /lustre/isaac/scratch/jtorre28/unstable/egapx/Ue_out -with-report /lustre/isaac/scratch/jtorre28/unstable/egapx/Ue_out/run.report.html -with-timeline /lustre/isaac/scratch/jtorre28/unstable/egapx/Ue_out/run.timeline.html -with-trace /lustre/isaac/scratch/jtorre28/unstable/egapx/Ue_out/run.trace.txt -params-file /lustre/isaac/scratch/jtorre28/unstable/egapx/Ue_out/run_params.yaml -resume

Seems to be an issue where gnomon has now output.

Here is the configuration file I used.

genome: /lustre/isaac/scratch/jtorre28/polishing/curated_no_bacteria.fasta
taxid: 87311

[pstrope]

Hi, Can you please confirm that you are using version v0.1.2-alpha from github?

[Artifice120]

Yes, I used the git clone command 4 days ago to download egapx .

[Artifice120]

Just ran the example data and this set ran fine.

[Artifice120]

I am using a node with 56 CPU and 2000 GB of RAM/Memory as well.

[pstrope]

Could you try with SRA data (using reads_ids option)? I see one, SRR1821979 (for Uroleucon ambrosiae). We think it is failing because it cannot find a protein set for the taxid you used, and the absence of SRA. Let me know how it goes. Thanks!

[Artifice120]

Thanks, will try that out.

Also the singularity option seems to work fine in a Slurm scheduler as long as you activate the Conda and virtulaenv environments in the sbatch file itself and cd out of the default path.

EDIT: Nextflow, pip, and Python had to be run in a conda environment saved in the same directory in seperate folders that the virtual-env was saved to to run in slurm without path errors. With the conda and virtual-env activated seperately

To save/create a conda env in a diffrent folder than the default use the following command format :

conda create -p path/to/wanted/env/location 

[Artifice120]

With the SRA_ids option this runs to completion.

[Artifice120]

Amazing, The raw predicted genes are in the same order of magnitude as what was expected (~17,000).

This anecdotally seems faster and more accurate than the Braker3 pipeline I have been using.

🎉🎉🎉

[murphyte]

Glad to hear that worked. Note we're still adding features to better characterize the annotation and add protein support for more organisms including aphids. So you'll want to run this again in a few months as the pipeline matures. More RNA-seq would be nice, but we do often see reasonable results with surprisingly small datasets.