maithermbarros
commented
2 months ago I was inspecting my files again and there seems to be a problem with my "search_space" and "region_to_gene_adj" files.
"search_space" only has a few genes in the list, which seems odd:
Consequently I have only a few unique targets and a total of 784 rows in region to gene adj. Here's how part of my region to gene adj file looks like:
I wonder what could have gone wrong when creating these files. I know that the files ACC_GEX.h5mu, genome_annotation.tsv and chromsizes.tsv are used to create search_space.tsv. These look ok to me, but maybe I am missing something?
Is there a guide somewhere that shows us what snakemake does step by step so I can try and traceback what has been done up to where I get an error? Thanks
Describe the bug When running scenicplus most of the pipeline works but when I get to eGRN_extended it fails. I already tried creating a new conda enviroment, reinstalling all packages, running snakemake in a brand new empty folder, but nothing works.
To Reproduce I prepared all files and ran snakemake within my conda environment
Error output Here's the full snakemake error log, but basically I get an ValueError saying "No objects to concatenate".
Expected behavior I checked the input files and they are all in the correct folder, correctly spelled and seem to be all fine.
Version (please complete the following information):
Additional context I tried running the code outside of snakemake, but it did not work either:
(scenicplus_3-11-8) maithe@kno01:/mnt/data/maithe/Gastric-Origins-Project/Gastric_Origin_Multiome/SCENICplus_P1-B1-B2P1-B2P3R2/scplus_pipeline_3rdtry_3118/Sn akemake$ scenicplus grn_inference eGRN --is_extended --TF_to_gene_adj_fname tf_to_gene_adj.tsv --region_to_gene_adj_fname region_to_gene_adj.tsv --cistromes_fname cistromes_extended.h5ad --ranking_db_fname /mnt/data/maithe/Gastric-Origins-Project/Gastric_Origin_Multiome/SCENICpl us_P1-B1-B2P1-B2P3R2/input/mm10_screen_v10_clust.regions_vs_motifs.rankings.feather --eRegulon_out_fname eRegulons_extended.tsv --tem p_dir /mnt/scratch/maithe/SCENICplus_temp_P1-B1-B2P1-B2P3R2/scplus_pipeline --order_regions_to_genes_by importance --order_TFs_to_gen es_by importance --gsea_n_perm 1000 --quantiles 0.85 0.90 0.95 --top_n_regionTogenes_per_gene 5 10 15 --top_n _regionTogenes_per_region --min_regions_per_gene 0 --rho_threshold 0.05 --min_target_genes 10 --n_cpu 10 2024-09-04 12:13:52,313 SCENIC+ INFO Loading TF to gene adjacencies. 2024-09-04 12:13:53,430 SCENIC+ INFO Loading region to gene adjacencies. 2024-09-04 12:13:53,433 SCENIC+ INFO Loading cistromes. 2024-09-04 12:13:53,605 GSEA INFO Thresholding region to gene relationships 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14/14 [00:36<00:00, 2.61s/it] 2024-09-04 12:14:30,349 GSEA INFO Subsetting TF2G adjacencies for TF with motif. 2024-09-04 12:14:30,476 GSEA INFO Running GSEA... 2024-09-04 12:14:30,476 GSEA INFO Generating rankings... 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 373/373 [00:01<00:00, 250.04it/s] 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 70/70 [00:00<00:00, 1975.24it/s] Running for Positive TF to gene: 0%| | 0/3865 [00:00<?, ?it/s] /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:71: RuntimeWarning: divide by zero encountered in divide norm_tag = 1.0/sum_correl_tag /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:74: RuntimeWarning: invalid value encountered in multiply RES = np.cumsum(tag_indicator * correl_vector * norm_tag - no_tag_indicator * norm_no_tag, axis=axis) Running for Positive TF to gene: 1%|▌ | 25/3865 [00:05<13:50, 4.62it/s] /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:71: RuntimeWarning: divide by zero encountered in divide norm_tag = 1.0/sum_correl_tag /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:74: RuntimeWarning: invalid value encountered in multiply RES = np.cumsum(tag_indicator * correl_vector * norm_tag - no_tag_indicator * norm_no_tag, axis=axis) /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:71: RuntimeWarning: divide by zero encountered in divide norm_tag = 1.0/sum_correl_tag /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:74: RuntimeWarning: invalid value encountered in multiply RES = np.cumsum(tag_indicator * correl_vector * norm_tag - no_tag_indicator * norm_no_tag, axis=axis) /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:71: RuntimeWarning: divide by zero encountered in divide norm_tag = 1.0/sum_correl_tag /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:74: RuntimeWarning: invalid value encountered in multiply RES = np.cumsum(tag_indicator * correl_vector * norm_tag - no_tag_indicator * norm_no_tag, axis=axis) /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:71: RuntimeWarning: divide by zero encountered in divide norm_tag = 1.0/sum_correl_tag /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:71: RuntimeWarning: divide by zero encountered in divide norm_tag = 1.0/sum_correl_tag /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:74: RuntimeWarning: invalid value encountered in multiply RES = np.cumsum(tag_indicator * correl_vector * norm_tag - no_tag_indicator * norm_no_tag, axis=axis) /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:74: RuntimeWarning: invalid value encountered in multiply RES = np.cumsum(tag_indicator * correl_vector * norm_tag - no_tag_indicator * norm_no_tag, axis=axis) /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:71: RuntimeWarning: divide by zero encountered in divide norm_tag = 1.0/sum_correl_tag /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:71: RuntimeWarning: divide by zero encountered in divide norm_tag = 1.0/sum_correl_tag /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:71: RuntimeWarning: divide by zero encountered in divide norm_tag = 1.0/sum_correl_tag /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:74: RuntimeWarning: invalid value encountered in multiply RES = np.cumsum(tag_indicator * correl_vector * norm_tag - no_tag_indicator * norm_no_tag, axis=axis) /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:74: RuntimeWarning: invalid value encountered in multiply RES = np.cumsum(tag_indicator * correl_vector * norm_tag - no_tag_indicator * norm_no_tag, axis=axis) /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:74: RuntimeWarning: invalid value encountered in multiply RES = np.cumsum(tag_indicator * correl_vector * norm_tag - no_tag_indicator * norm_no_tag, axis=axis) /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:71: RuntimeWarning: divide by zero encountered in divide norm_tag = 1.0/sum_correl_tag /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:74: RuntimeWarning: invalid value encountered in multiply RES = np.cumsum(tag_indicator * correl_vector * norm_tag - no_tag_indicator * norm_no_tag, axis=axis) /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/joblib/externals/loky/process_executor.py:752: UserWarning: A worker stopped whil e some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak. warnings.warn( /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:71: RuntimeWarning: divide by zero encountered in divide norm_tag = 1.0/sum_correl_tag /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:74: RuntimeWarning: invalid value encountered in multiply RES = np.cumsum(tag_indicator * correl_vector * norm_tag - no_tag_indicator * norm_no_tag, axis=axis) /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:71: RuntimeWarning: divide by zero encountered in divide norm_tag = 1.0/sum_correl_tag /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:74: RuntimeWarning: invalid value encountered in multiply RES = np.cumsum(tag_indicator * correl_vector * norm_tag - no_tag_indicator * norm_no_tag, axis=axis) Running for Positive TF to gene: 3%|██▎ | 108/3865 [00:13<05:05, 12.30it/s] /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:71: RuntimeWarning: divide by zero encountered in divide norm_tag = 1.0/sum_correl_tag /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:74: RuntimeWarning: invalid value encountered in multiply RES = np.cumsum(tag_indicator * correl_vector * norm_tag - no_tag_indicator * norm_no_tag, axis=axis) Running for Positive TF to gene: 5%|███▉ | 182/3865 [00:13<01:54, 32.16it/s] /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:71: RuntimeWarning: divide by zero encountered in divide norm_tag = 1.0/sum_correl_tag /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:74: RuntimeWarning: invalid value encountered in multiply RES = np.cumsum(tag_indicator * correl_vector * norm_tag - no_tag_indicator * norm_no_tag, axis=axis) Running for Positive TF to gene: 9%|███████▋ | 353/3865 [00:14<00:40, 85.97it/s] /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:71: RuntimeWarning: divide by zero encountered in divide norm_tag = 1.0/sum_correl_tag /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:74: RuntimeWarning: invalid value encountered in multiply RES = np.cumsum(tag_indicator * correl_vector * norm_tag - no_tag_indicator * norm_no_tag, axis=axis) Running for Positive TF to gene: 15%|████████████▎ | 571/3865 [00:15<00:22, 144.34it/s] /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:71: RuntimeWarning: divide by zero encountered in divide norm_tag = 1.0/sum_correl_tag /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:74: RuntimeWarning: invalid value encountered in multiply RES = np.cumsum(tag_indicator * correl_vector * norm_tag - no_tag_indicator * norm_no_tag, axis=axis) Running for Positive TF to gene: 20%|████████████████▌ | 774/3865 [00:16<00:16, 190.31it/s] /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:71: RuntimeWarning: divide by zero encountered in divide norm_tag = 1.0/sum_correl_tag /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:74: RuntimeWarning: invalid value encountered in multiply RES = np.cumsum(tag_indicator * correl_vector * norm_tag - no_tag_indicator * norm_no_tag, axis=axis) Running for Positive TF to gene: 23%|██████████████████▋ | 872/3865 [00:17<00:15, 196.12it/s] /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:71: RuntimeWarning: divide by zero encountered in divide norm_tag = 1.0/sum_correl_tag Running for Positive TF to gene: 25%|█████████████████████ | 981/3865 [00:17<00:12, 236.39it/s] /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:74: RuntimeWarning: invalid value encountered in multiply RES = np.cumsum(tag_indicator * correl_vector * norm_tag - no_tag_indicator * norm_no_tag, axis=axis) Running for Positive TF to gene: 31%|█████████████████████████▌ | 1203/3865 [00:18<00:09, 267.47it/s] /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:71: RuntimeWarning: divide by zero encountered in divide norm_tag = 1.0/sum_correl_tag /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:74: RuntimeWarning: invalid value encountered in multiply RES = np.cumsum(tag_indicator * correl_vector * norm_tag - no_tag_indicator * norm_no_tag, axis=axis) Running for Positive TF to gene: 40%|████████████████████████████████▉ | 1551/3865 [00:18<00:06, 347.74it/s] /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:71: RuntimeWarning: divide by zero encountered in divide norm_tag = 1.0/sum_correl_tag /home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/gseapy/algorithm.py:74: RuntimeWarning: invalid value encountered in multiply RES = np.cumsum(tag_indicator * correl_vector * norm_tag - no_tag_indicator * norm_no_tag, axis=axis) Running for Positive TF to gene: 100%|██████████████████████████████████████████████████████████████████████████████████| 3865/3865 [00:25<00:00, 154.38it/s] Running for Negative TF to gene: 100%|████████████████████████████████████████████████████████████████████████████████| 3865/3865 [00:00<00:00, 12600.68it/s] 2024-09-04 12:14:58,996 GSEA INFO Subsetting on adjusted pvalue: 1, minimal NES: 0 and minimal leading edge genes 10 2024-09-04 12:14:58,997 GSEA INFO Merging eRegulons 2024-09-04 12:14:59,039 SCENIC+ INFO Formatting eGRN as table. Traceback (most recent call last): File "/home/maithe/miniconda3/envs/scenicplus_3-11-8/bin/scenicplus", line 8, in <module> sys.exit(main()) ^^^^^^ File "/home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/scenicplus/cli/scenicplus.py", line 1137, in main args.func(args) File "/home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/scenicplus/cli/scenicplus.py", line 861, in eGRN infer_grn( File "/home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/scenicplus/cli/commands.py", line 912, in infer_grn eRegulon_metadata = _format_egrns( ^^^^^^^^^^^^^^ File "/home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/scenicplus/cli/commands.py", line 790, in _format_egrns eRegulon_metadata = pd.concat(eRegulons_formatted) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/pandas/util/_decorators.py", line 317, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/pandas/core/reshape/concat.py", line 369, in concat op = _Concatenator( ^^^^^^^^^^^^^^ File "/home/maithe/miniconda3/envs/scenicplus_3-11-8/lib/python3.11/site-packages/pandas/core/reshape/concat.py", line 426, in __init__ raise ValueError("No objects to concatenate") ValueError: No objects to concatenateAny help would be greatly appreciated. Thanks a lot!