Error in rule call_merge_haplotypes_batch

[Milia1368]

Hello, I encountered into some problems while using this tool from Docker to call SV from assembly-hap and I tried many times and still couldn't make it. If you can give me some suggestions, I'd appreciate it! The commands and configuration files are listed below:

assemblies.tsv NAME HAP1 HAP2 NA24385 /path/v12_NA24385_hpg_pbsq2-ccs_1000-pereg.h1-un.racon-p2.fasta /path/v12_NA24385_hpg_pbsq2-ccs_1000-pereg.h2-un.racon-p2.fasta config.json { "reference": "/path/GRCh38_full_analysis_set_plus_decoy_hla.fa" }

\time -v singularity run --bind "/data,$(pwd):$(pwd)" library://becklab/pav/pav:latest -c 16 > run.log 2>&1 & And part of run.log is as follows: The begain:

Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 16
Rules claiming more threads will be scaled down.
Job stats:
job                             count    min threads    max threads

----------------------------  -------  -------------  -------------

align_cut_tig_overlap               2              1              1
align_get_read_bed                  2              1              1
align_get_tig_fa                    2              1              1
align_map                           2             12             12
call_cigar                         20              1              1
call_cigar_merge                    2              1              1
call_final_bed                      1              1              1
call_integrate_sources              2              1              1
call_inv_batch                    120              4              4
call_inv_batch_merge                2              1              1
call_inv_cluster                    4              1              1
call_inv_flag_insdel_cluster        4              1              1
call_inv_merge_flagged_loci         2              1              1
call_lg_discover                   20             12             12
call_lg_split                       2              1              1
call_mappable_bed                   2              1              1
call_merge_batch_table              1              1              1
call_merge_haplotypes               4              1              1
call_merge_haplotypes_batch        80             12             12
call_merge_lg                       6              1              1
data_align_ref                      1              1              1
data_align_ref_anno_n_gap           1              1              1
data_ref_contig_table               1              1              1
pav_all                             1              1              1
vcf_write_vcf                       1              1              1
total                             285              1             12

Select jobs to execute... 

Error:

[Tue Mar 26 02:55:58 2024]
Error in rule call_merge_haplotypes_batch:
    jobid: 0
    input: data/ref/merge_batch.tsv.gz, results/NA24385/bed/pre_merge/h1/snv_snv.bed.gz, results/NA24385/bed/pre_merge/h2/snv_snv.bed.gz, results/NA24385/callable/callable_regions_h1_500.bed.gz, results/NA24385/callable/callable_regions_h2_500.bed.gz
    output: temp/NA24385/bed/bychrom/snv_snv/5.bed.gz

> RuleException:

KeyError in line 172 of /opt/pav/rules/call.snakefile:
"['chr6-31694708-SNV-T', 'chr6-31694712-SNV-C', 'chr6-31694727-SNV-A', 'chr6-31694731-SNV-C'] not in index"
  File "/opt/pav/rules/call.snakefile", line 172, in __rule_call_merge_haplotypes_batch
  File "/opt/pav/pavlib/call.py", line 283, in merge_haplotypes
  File "/opt/pav/dep/svpop/svpoplib/svmerge.py", line 78, in merge_variants
  File "/opt/pav/dep/svpop/svpoplib/svmerge.py", line 513, in merge_variants_nr
  File "/opt/pav/dep/svpop/svpoplib/svmerge.py", line 712, in merge_sample_by_support
  File "/usr/local/lib/python3.11/site-packages/pandas/core/indexing.py", line 1073, in __getitem__
  File "/usr/local/lib/python3.11/site-packages/pandas/core/indexing.py", line 1301, in _getitem_axis
  File "/usr/local/lib/python3.11/site-packages/pandas/core/indexing.py", line 1239, in _getitem_iterable
  File "/usr/local/lib/python3.11/site-packages/pandas/core/indexing.py", line 1432, in _get_listlike_indexer
  File "/usr/local/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 6113, in _get_indexer_strict
  File "/usr/local/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 6176, in _raise_if_missing
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Select jobs to execute...

[Tue Mar 26 02:55:58 2024]
rule call_merge_haplotypes_batch:

The final:

[Tue Mar 26 08:39:03 2024]
Finished job 243.
280 of 285 steps (98%) done
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-03-25T093812.287047.snakemake.log
Command exited with non-zero status 1
        Command being timed: "singularity run --bind /data,$(pwd):$(pwd) library://becklab/pav/pav:latest -c 16"
        User time (seconds): 109042.70
        System time (seconds): 17104.72
        Percent of CPU this job got: 152%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 23:01:01
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 28827968
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 3654
        Minor (reclaiming a frame) page faults: 4936269076
        Voluntary context switches: 16937804
        Involuntary context switches: 6188440463
        Swaps: 0
        File system inputs: 8893868
        File system outputs: 7096544
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 1
(END)

Thanks!

[paudano]

My apologies, I missed this issue. The SNV IDs don't look right, and the base at the sites is the reference base ("T" is the reference allele at chr6:31694708 in GRCh38). Does this work with a later version of PAV? If so, send me the error and I'll track it down.

[Milia1368]

I was using the latest version on docker, which is v2.3.4. To use your latest 2.4.1 code, I need to run it from code. But there is no runlocal nor rundist under the pav directory, which are written in the NATIVE_INSTALL.md. Seems there are some changes. How am I supposed to run the latest code?Any advice you can offer would be greatly appreciated.

[paudano]

2.4.1 is on Docker: https://hub.docker.com/r/becklab/pav/tags

You should be able to run this from Docker.

If you haven't run a native version of PAV, you'd have to install a whole environment with Python, Snakemake, and dependencies and run snakemake directly. The rundist and runlocal went away recently, they are no longer needed. Instead, a profile is distributed with PAV, which can be run with Snakemake directly. For example, to distribute over Slurm (20 simultaneous jobs): snakemake -j 20 --profile profiles/default --executor slurm. This requires Snakemake with the Slurm executor installed.

[Milia1368]

Turns out I was using singularity library instead of docker hub, BTW, the singularity library seems has the version 2.3.4 as the latest. But there are still problems. Whether I use singularity to run the 2.4.1 image on docker hub, or I run docker to run the 2.4.1 image, I get the same error:

ModuleNotFoundError in file /opt/pav/Snakefile, line 53:
No module named 'joblib'
  File "/opt/pav/Snakefile", line 53, in <module>
  File "/opt/pav/pavlib/__init__.py", line 7, in <module>
  File "/opt/pav/pavlib/align/__init__.py", line 3, in <module>
  File "/opt/pav/pavlib/align/align.py", line 6, in <module>
  File "/opt/pav/pavlib/seq.py", line 16, in <module>
  File "/opt/pav/dep/svpop/svpoplib/__init__.py", line 24, in <module>
  File "/opt/pav/dep/svpop/svpoplib/svmerge.py", line 7, in <module>

Furthermore, FYI, if I run docker instead of singularity, I need to bind another volume "/.cache" to the container to get the image running:

mkdir .cache
sudo docker run --rm -v ${PWD}:${PWD} -v ${PWD}/.cache:/.cache --user "$(id -u):$(id -g)" --workdir ${PWD} becklab/pav:latest -c 16

Could you offer any advice? I would be greatly appreciated.Thanks!

[paudano]

I need to build joblib into the container, working on that this week.

[paudano]

Testing now, it will be released in v2.4.2 once tests complete. I'll update this ticket when it's available.

[paudano]

I am temporarily releasing 2.4.1.2. I want to do more testing before I call it a stable release (any PAV version with a fourth number is a development version). It's out now if you want to try it. I expect 2.4.2 this week or next (it's going to be a busy week).