search

aertslab / scenicplus

SCENIC+ is a python package to build gene regulatory networks (GRNs) using combined or separate single-cell gene expression (scRNA-seq) and single-cell chromatin accessibility (scATAC-seq) data.
Other
186 stars 29 forks source link

SCENIC+ crashed in tf_to_gene step #454

Open kangjiajinlong opened 2 months ago

kangjiajinlong commented 2 months ago

Dear SCENIC+ team,

I am using SCENIC+ and it crashed in tf_to_gene step. The error log is as follows:

Assuming unrestricted shared filesystem usage for local execution.
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 4
Rules claiming more threads will be scaled down.
Job stats:
job                count
---------------  -------
AUCell_direct          1
AUCell_extended        1
all                    1
eGRN_direct            1
eGRN_extended          1
scplus_mudata          1
tf_to_gene             1
total                  7

Select jobs to execute...
Execute 1 jobs...

[Wed Aug 21 13:30:19 2024]
localrule tf_to_gene:
    input: /home/jkang5/sc163/scenicplus/outs/ACC_GEX.h5mu, /home/jkang5/sc163/scenicplus/outs/tf_names.txt
    output: /home/jkang5/sc163/scenicplus/outs/tf_to_gene_adj.tsv
    jobid: 5
    reason: Missing output files: /home/jkang5/sc163/scenicplus/outs/tf_to_gene_adj.tsv
    threads: 4
    resources: tmpdir=/tmp

2024-08-21 13:31:00,298 SCENIC+      INFO     Reading multiome MuData.
/rsrch5/home/sct-rsch/jkang5/venvs/scenicplus/lib/python3.11/site-packages/anndata/_core/anndata.py:522: FutureWarning: The dtype argument is deprecated and will be removed in late 2024.
  warnings.warn(
/rsrch5/home/sct-rsch/jkang5/venvs/scenicplus/lib/python3.11/site-packages/anndata/_core/anndata.py:522: FutureWarning: The dtype argument is deprecated and will be removed in late 2024.
  warnings.warn(
2024-08-21 13:31:36,668 SCENIC+      INFO     Using 747 TFs.
2024-08-21 13:31:37,073 TF2G         INFO     Calculating TF-to-gene importance

Running using 4 cores:   0%|          | 0/27636 [00:00<?, ?it/s]
Running using 4 cores:   0%|          | 4/27636 [00:00<40:53, 11.26it/s]
Running using 4 cores:   0%|          | 4/27636 [00:19<40:53, 11.26it/s]
...
Running using 4 cores: 100%|█████████▉| 27632/27636 [62:57:00<00:21,  5.38s/it]
Running using 4 cores: 100%|██████████| 27636/27636 [62:57:20<00:00,  5.30s/it]
Running using 4 cores: 100%|██████████| 27636/27636 [62:57:20<00:00,  8.20s/it]
2024-08-24 04:29:27,730 TF2G         INFO     Adding correlation coefficients to adjacencies.
[Sat Aug 24 04:29:41 2024]
Error in rule tf_to_gene:
    jobid: 5
    input: /home/jkang5/sc163/scenicplus/outs/ACC_GEX.h5mu, /home/jkang5/sc163/scenicplus/outs/tf_names.txt
    output: /home/jkang5/sc163/scenicplus/outs/tf_to_gene_adj.tsv
    shell:

        scenicplus grn_inference TF_to_gene             --multiome_mudata_fname /home/jkang5/sc163/scenicplus/outs/ACC_GEX.h5mu             --tf_names /home/jkang5/sc163/scenicplus/outs/tf_names.txt             --temp_dir /home/jkang5/sc163/scenicplus/temp             --out_tf_to_gene_adjacencies /home/jkang5/sc163/scenicplus/outs/tf_to_gene_adj.tsv             --method GBM             --n_cpu 4             --seed 666

        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-08-21T133013.885952.snakemake.log
WorkflowError:
At least one job did not complete successfully.
/rsrch5/home/sct-rsch/jkang5/venvs/scenicplus/lib/python3.11/site-packages/joblib/externals/loky/backend/resource_tracker.py:314: UserWarning: resource_tracker: There appear to be 10 leaked semlock objects to clean up at shutdown
  warnings.warn(
/rsrch5/home/sct-rsch/jkang5/venvs/scenicplus/lib/python3.11/site-packages/joblib/externals/loky/backend/resource_tracker.py:314: UserWarning: resource_tracker: There appear to be 1 leaked folder objects to clean up at shutdown
  warnings.warn(
/rsrch5/home/sct-rsch/jkang5/venvs/scenicplus/lib/python3.11/site-packages/joblib/externals/loky/backend/resource_tracker.py:330: UserWarning: resource_tracker: /home/jkang5/sc163/scenicplus/temp/joblib_memmapping_folder_3191096_65ae4f6fde3149ff8e587d14176cb8c4_e980a3c3e49b4ce9bf7e2b107c50d27a: FileNotFoundError(2, 'No such file or directory')
  warnings.warn(f"resource_tracker: {name}: {e!r}")

Could you kindly help resolve this issue?

Thanks, Jack

SeppeDeWinter commented 2 months ago

Hi @kangjiajinlong

Could you share the log file? .snakemake/log/2024-08-21T133013.885952.snakemake.log

All the best,

Seppe

kangjiajinlong commented 2 months ago

This is the full content of that file:

Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 4 Rules claiming more threads will be scaled down. Job stats: job count

AUCell_direct 1 AUCell_extended 1 all 1 eGRN_direct 1 eGRN_extended 1 scplus_mudata 1 tf_to_gene 1 total 7

Select jobs to execute... Execute 1 jobs...

[Wed Aug 21 13:30:19 2024] localrule tf_to_gene: input: /home/jkang5/sc163/scenicplus/outs/ACC_GEX.h5mu, /home/jkang5/sc163/scenicplus/outs/tf_names.txt output: /home/jkang5/sc163/scenicplus/outs/tf_to_gene_adj.tsv jobid: 5 reason: Missing output files: /home/jkang5/sc163/scenicplus/outs/tf_to_gene_adj.tsv threads: 4 resources: tmpdir=/tmp

[Sat Aug 24 04:29:41 2024] Error in rule tf_to_gene: jobid: 5 input: /home/jkang5/sc163/scenicplus/outs/ACC_GEX.h5mu, /home/jkang5/sc163/scenicplus/outs/tf_names.txt output: /home/jkang5/sc163/scenicplus/outs/tf_to_gene_adj.tsv shell:

    scenicplus grn_inference TF_to_gene             --multiome_mudata_fname /home/jkang5/sc163/scenicplus/outs/ACC_GEX.h5mu             --tf_names /home/jkang5/sc163/scenicplus/outs/tf_names.txt             --temp_dir /home/jkang5/sc163/scenicplus/temp             --out_tf_to_gene_adjacencies /home/jkang5/sc163/scenicplus/outs/tf_to_gene_adj.tsv             --method GBM             --n_cpu 4             --seed 666

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2024-08-21T133013.885952.snakemake.log WorkflowError: At least one job did not complete successfully.

SeppeDeWinter commented 2 months ago

Hi @kangjiajinlong

Hmm.. that's still not a nice stack trace (i.e., not easy to tell from this what is going wrong).

Could you try running the command manually to see wether you get any error


scenicplus grn_inference TF_to_gene \
            --multiome_mudata_fname /home/jkang5/sc163/scenicplus/outs/ACC_GEX.h5mu \
            --tf_names /home/jkang5/sc163/scenicplus/outs/tf_names.txt  \
           --temp_dir /home/jkang5/sc163/scenicplus/temp \
            --out_tf_to_gene_adjacencies /home/jkang5/sc163/scenicplus/outs/tf_to_gene_adj.tsv \
            --method GBM \
            --n_cpu 4  \
           --seed 666

Best,

Seppe