Open zhangdong360 opened 5 days ago
My code:
#!/bin/bash
#SBATCH -o output/pyscenic_hsc_sev.out
#SBATCH -e output/pyscenic_hsc_sev.err
#SBATCH --partition=compute
#SBATCH -J scenic_HSC_SEV
#SBATCH --nodes=1
#SBATCH -n 30
# This is for fastp protocol
#conda activate scanpy
# human
#f_db_names="/share/home/zhangd/tools/database/cistarget/cisTarget_databases/homo_sapiens/hg38/refseq_r80/mc_v10_clust/gene_based/hg38_500bp_up_100bp_down_full_tx_v10_clust.genes_vs_motifs.rankings.feather"
#f_motif_path="/share/home/zhangd/tools/database/cistarget/Motif2TF/motifs-v10nr_clust-nr.hgnc-m0.001-o0.0.tbl"
#f_tf_list="/share/home/zhangd/project/python_project/pySCENIC/allTFs_hg38.txt"
# mouse
f_db_names="/home/zhangdong_2/database/cistarget/cisTarget_databases/mus_musculus/mm10/refseq_r80/mc_v10_clust/mm10_500bp_up_100bp_down_full_tx_v10_clust.genes_vs_motifs.rankings.feather"
f_motif_path="/home/zhangdong_2/database/cistarget/Motif2TF/motifs-v10nr_clust-nr.mgi-m0.001-o0.0.tbl"
f_tf_list="/home/zhangdong_2/database/cistarget/TF_lists/allTFs_mm.txt"
# data input
dir_result="/home/zhangdong_2/project/pySCENIC/03_result/HSC_SEV/"
input_loom="/home/zhangdong_2/project/pySCENIC/01_data/HSC_SEV.loom"
# step1
echo "Step 1 pyscenic grn start"
nohup pyscenic grn ${input_loom} ${f_tf_list} \
--seed 21 \
--num_workers 16 \
--method grnboost2 \
--output ${dir_result}/step_1_fibo_grn.tsv >step1.out 2>&1 &
echo "Step 1 pyscenic grn finish"
echo "Step 2 pyscenic ctx start"
nohup pyscenic ctx ${dir_result}/step_1_fibo_grn.tsv \
${f_db_names} \
--annotations_fname ${f_motif_path} \
--expression_mtx_fname ${input_loom} \
--output ${dir_result}/step_2_reg.csv \
--mask_dropouts \
--num_workers 16 >step2.out 2>&1 &
echo "Step 2 pyscenic ctx finish"
echo "Step 3 pyscenic aucell start"
pyscenic aucell \
${input_loom} \
${dir_result}/step_2_reg.csv \
--seed 21 \
--output ${dir_result}/step_3_aucell.csv \
--num_workers 16 >step_3.out 2>&1 &
echo "All finish"
I tried to use slurm distribution to compute nodes and directly in local bash operation, the result is the same.
At the same time I run this code on the other operation platform, I found warning almost consistent, all point to the ndarray - a612cf0abd06497fa68e9db39636fedb
.
May be some help to found the problem?
My code:
#!/bin/bash #SBATCH -o output/pyscenic_hsc_sev.out #SBATCH -e output/pyscenic_hsc_sev.err #SBATCH --partition=compute #SBATCH -J scenic_HSC_SEV #SBATCH --nodes=1 #SBATCH -n 30 # This is for fastp protocol #conda activate scanpy # human #f_db_names="/share/home/zhangd/tools/database/cistarget/cisTarget_databases/homo_sapiens/hg38/refseq_r80/mc_v10_clust/gene_based/hg38_500bp_up_100bp_down_full_tx_v10_clust.genes_vs_motifs.rankings.feather" #f_motif_path="/share/home/zhangd/tools/database/cistarget/Motif2TF/motifs-v10nr_clust-nr.hgnc-m0.001-o0.0.tbl" #f_tf_list="/share/home/zhangd/project/python_project/pySCENIC/allTFs_hg38.txt" # mouse f_db_names="/home/zhangdong_2/database/cistarget/cisTarget_databases/mus_musculus/mm10/refseq_r80/mc_v10_clust/mm10_500bp_up_100bp_down_full_tx_v10_clust.genes_vs_motifs.rankings.feather" f_motif_path="/home/zhangdong_2/database/cistarget/Motif2TF/motifs-v10nr_clust-nr.mgi-m0.001-o0.0.tbl" f_tf_list="/home/zhangdong_2/database/cistarget/TF_lists/allTFs_mm.txt" # data input dir_result="/home/zhangdong_2/project/pySCENIC/03_result/HSC_SEV/" input_loom="/home/zhangdong_2/project/pySCENIC/01_data/HSC_SEV.loom" # step1 echo "Step 1 pyscenic grn start" nohup pyscenic grn ${input_loom} ${f_tf_list} \ --seed 21 \ --num_workers 16 \ --method grnboost2 \ --output ${dir_result}/step_1_fibo_grn.tsv >step1.out 2>&1 & echo "Step 1 pyscenic grn finish" echo "Step 2 pyscenic ctx start" nohup pyscenic ctx ${dir_result}/step_1_fibo_grn.tsv \ ${f_db_names} \ --annotations_fname ${f_motif_path} \ --expression_mtx_fname ${input_loom} \ --output ${dir_result}/step_2_reg.csv \ --mask_dropouts \ --num_workers 16 >step2.out 2>&1 & echo "Step 2 pyscenic ctx finish" echo "Step 3 pyscenic aucell start" pyscenic aucell \ ${input_loom} \ ${dir_result}/step_2_reg.csv \ --seed 21 \ --output ${dir_result}/step_3_aucell.csv \ --num_workers 16 >step_3.out 2>&1 & echo "All finish"
I tried to use slurm distribution to compute nodes and directly in local bash operation, the result is the same. At the same time I run this code on the other operation platform, I found warning almost consistent, all point to the
ndarray - a612cf0abd06497fa68e9db39636fedb
. May be some help to found the problem?
NOTE:It is worth mentioning that this is a different data on different platforms have been the same warning…And my sample book set can be successful operation.This doubt has been gnawed at me for a long time.
Can you with try with the Docker/Podman/Singularity/Apptainer images instead? https://pyscenic.readthedocs.io/en/latest/installation.html#docker-podman-and-singularity-apptainer-images
Describe the bug When I run pySCENIC, I often encounter disturbing warnings. I checked the problem may be associated with me this question. https://github.com/aertslab/pySCENIC/issues/482 But I'm not using port 8787. On the other hand, I don't often encounter this warning on the HPC where I have Rstudio server installed, and I don't think it has anything to do with it. I think the problem might be with dask, but I'm not well versed in it. On the other hand, the lack of output, which makes me cannot judge whether I need to run the program. As mentioned above, re-running the program will most likely encounter warning again. In addition, I have tried arboreto_with_multiprocessing.py, but it was too inefficient, I tested it on small samples, and it was nearly twice as slow as pySCENIC for the same number of CPU cores. I don't think that's acceptable in a large sample. It took me too much energy in to run the program, I have to sample cut to my data size, but I don't think this is a long-term solution.
Expected behavior I didn't find a clear reproduction. But I find it often will appear in my after a run.