ab4cp
commented
10 months ago I attempted to run it again with the snakemake pathway. Not surprisingly I got the same error. Here's the script below which might be more helpful than the one I previously posted.
(scrna-workflow) unam@IP:~/scrna-workflow$ snakemake -j 10 --config datafolder=data option=minimal {'10X_17_028'} Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 10 Rules claiming more threads will be scaled down. Job stats: job count
all 1 normalization_pca_rds 1 plot_dimplots 3 plot_selected_marker_plots_separately 2 plot_singler_celltype 1 plot_singler_dimplots 3 plot_some_metrics 1 plot_some_technicals 1 singler_annotation 1 total 14
Select jobs to execute...
[Sat Nov 4 11:35:23 2023] rule normalization_pca_rds: input: analyses/raw/percent_mt~10/resolution~0.8/10X_17_028.rds output: analyses/processed/percent_mt~10/resolution~0.8/10X_17_028.rds jobid: 2 reason: Missing output files: analyses/processed/percent_mt~10/resolution~0.8/10X_17_028.rds wildcards: sample=10X_17_028, percent_mt=10, resolution=0.8 threads: 5 resources: tmpdir=/tmp, mem_mb=5000, mem_mib=4769
Loading required package: optparse Warning message: package ‘optparse’ was built under R version 4.2.3 Loading required package: SingleR Loading required package: SummarizedExperiment Loading required package: MatrixGenerics Loading required package: matrixStats
Attaching package: ‘MatrixGenerics’
The following objects are masked from ‘package:matrixStats’:
colAlls, colAnyNAs, colAnys, colAvgsPerRowSet, colCollapse,
colCounts, colCummaxs, colCummins, colCumprods, colCumsums,
colDiffs, colIQRDiffs, colIQRs, colLogSumExps, colMadDiffs,
colMads, colMaxs, colMeans2, colMedians, colMins, colOrderStats,
colProds, colQuantiles, colRanges, colRanks, colSdDiffs, colSds,
colSums2, colTabulates, colVarDiffs, colVars, colWeightedMads,
colWeightedMeans, colWeightedMedians, colWeightedSds,
colWeightedVars, rowAlls, rowAnyNAs, rowAnys, rowAvgsPerColSet,
rowCollapse, rowCounts, rowCummaxs, rowCummins, rowCumprods,
rowCumsums, rowDiffs, rowIQRDiffs, rowIQRs, rowLogSumExps,
rowMadDiffs, rowMads, rowMaxs, rowMeans2, rowMedians, rowMins,
rowOrderStats, rowProds, rowQuantiles, rowRanges, rowRanks,
rowSdDiffs, rowSds, rowSums2, rowTabulates, rowVarDiffs, rowVars,
rowWeightedMads, rowWeightedMeans, rowWeightedMedians,
rowWeightedSds, rowWeightedVarsLoading required package: GenomicRanges Loading required package: stats4 Loading required package: BiocGenerics
Attaching package: ‘BiocGenerics’
The following objects are masked from ‘package:stats’:
IQR, mad, sd, var, xtabsThe following objects are masked from ‘package:base’:
anyDuplicated, aperm, append, as.data.frame, basename, cbind,
colnames, dirname, do.call, duplicated, eval, evalq, Filter, Find,
get, grep, grepl, intersect, is.unsorted, lapply, Map, mapply,
match, mget, order, paste, pmax, pmax.int, pmin, pmin.int,
Position, rank, rbind, Reduce, rownames, sapply, setdiff, sort,
table, tapply, union, unique, unsplit, which.max, which.minLoading required package: S4Vectors
Attaching package: ‘S4Vectors’
The following objects are masked from ‘package:base’:
expand.grid, I, unnameLoading required package: IRanges Loading required package: GenomeInfoDb Loading required package: Biobase Welcome to Bioconductor
Vignettes contain introductory material; view with
'browseVignettes()'. To cite Bioconductor, see
'citation("Biobase")', and for packages 'citation("pkgname")'.Attaching package: ‘Biobase’
The following object is masked from ‘package:MatrixGenerics’:
rowMediansThe following objects are masked from ‘package:matrixStats’:
anyMissing, rowMediansWarning messages: 1: package ‘SingleR’ was built under R version 4.2.3 2: package ‘matrixStats’ was built under R version 4.2.3 3: package ‘GenomicRanges’ was built under R version 4.2.3 4: package ‘S4Vectors’ was built under R version 4.2.3 5: package ‘IRanges’ was built under R version 4.2.3 6: package ‘GenomeInfoDb’ was built under R version 4.2.3 7: package ‘Biobase’ was built under R version 4.2.3 Loading required package: celldex
Attaching package: ‘celldex’
The following objects are masked from ‘package:SingleR’:
BlueprintEncodeData, DatabaseImmuneCellExpressionData,
HumanPrimaryCellAtlasData, ImmGenData, MonacoImmuneData,
MouseRNAseqData, NovershternHematopoieticDataLoading required package: tidyverse ── Attaching packages ─────────────────────────────────────────────── tidyverse 1.3.2 ── ✔ ggplot2 3.4.4 ✔ purrr 1.0.2 ✔ tibble 3.2.1 ✔ dplyr 1.1.3 ✔ tidyr 1.3.0 ✔ stringr 1.5.0 ✔ readr 2.1.4 ✔ forcats 1.0.0 ── Conflicts ────────────────────────────────────────────────── tidyverse_conflicts() ── ✖ dplyr::collapse() masks IRanges::collapse() ✖ dplyr::combine() masks Biobase::combine(), BiocGenerics::combine() ✖ dplyr::count() masks matrixStats::count() ✖ dplyr::desc() masks IRanges::desc() ✖ tidyr::expand() masks S4Vectors::expand() ✖ dplyr::filter() masks stats::filter() ✖ dplyr::first() masks S4Vectors::first() ✖ dplyr::lag() masks stats::lag() ✖ ggplot2::Position() masks BiocGenerics::Position(), base::Position() ✖ purrr::reduce() masks GenomicRanges::reduce(), IRanges::reduce() ✖ dplyr::rename() masks S4Vectors::rename() ✖ dplyr::slice() masks IRanges::slice() Warning messages: 1: package ‘ggplot2’ was built under R version 4.2.3 2: package ‘tibble’ was built under R version 4.2.3 3: package ‘tidyr’ was built under R version 4.2.3 4: package ‘readr’ was built under R version 4.2.3 5: package ‘purrr’ was built under R version 4.2.3 6: package ‘dplyr’ was built under R version 4.2.3 7: package ‘stringr’ was built under R version 4.2.3 8: package ‘forcats’ was built under R version 4.2.3 Loading required package: Seurat Attaching SeuratObject Seurat v4 was just loaded with SeuratObject v5; disabling v5 assays and validation routines, and ensuring assays work in strict v3/v4 compatibility mode
Attaching package: ‘Seurat’
The following object is masked from ‘package:SummarizedExperiment’:
AssaysWarning message:
package ‘Seurat’ was built under R version 4.2.3
Loading required package: patchwork
Warning message:
package ‘patchwork’ was built under R version 4.2.3
Traceback (most recent call last):
File "
Number of nodes: 6809 Number of edges: 239085
Running Louvain algorithm... 0% 10 20 30 40 50 60 70 80 90 100% [----|----|----|----|----|----|----|----|----|----| **| Maximum modularity in 10 random starts: 0.9127 Number of communities: 23 Elapsed time: 0 seconds Warning: The default method for RunUMAP has changed from calling Python UMAP via reticulate to the R-native UWOT using the cosine metric To use Python UMAP via reticulate, set umap.method to 'umap-learn' and metric to 'correlation' This message will be shown once per session 11:35:47 UMAP embedding parameters a = 0.9922 b = 1.112 Found more than one class "dist" in cache; using the first, from namespace 'BiocGenerics' Also defined by ‘spam’ 11:35:47 Read 6809 rows and found 23 numeric columns 11:35:47 Using Annoy for neighbor search, n_neighbors = 30 Found more than one class "dist" in cache; using the first, from namespace 'BiocGenerics' Also defined by ‘spam’ 11:35:47 Building Annoy index with metric = cosine, n_trees = 50 0% 10 20 30 40 50 60 70 80 90 100% [----|----|----|----|----|----|----|----|----|----| **| 11:35:47 Writing NN index file to temp file /tmp/RtmpDyhI2v/file38dab932e3acf7 11:35:47 Searching Annoy index using 1 thread, search_k = 3000 11:35:49 Annoy recall = 100% 11:35:50 Commencing smooth kNN distance calibration using 1 thread with target n_neighbors = 30 11:35:51 Initializing from normalized Laplacian + noise (using irlba) 11:35:52 Commencing optimization for 500 epochs, with 280188 positive edges Using method 'umap' 0% 10 20 30 40 50 60 70 80 90 100% [----|----|----|----|----|----|----|----|----|----| **| 11:36:00 Optimization finished Loading required package: DoubletFinder Loading required package: fields Loading required package: spam Spam version 2.10-0 (2023-10-23) is loaded. Type 'help( Spam)' or 'demo( spam)' for a short introduction and overview of this package. Help for individual functions is also obtained by adding the suffix '.spam' to the function name, e.g. 'help( chol.spam)'.
Attaching package: ‘spam’
The following object is masked from ‘package:stats4’:
mleThe following objects are masked from ‘package:base’:
backsolve, forwardsolveLoading required package: viridisLite
Try help(fields) to get started.
Loading required package: KernSmooth
KernSmooth 2.23 loaded
Copyright M. P. Wand 1997-2009
[1] "Creating 2270 artificial doublets..."
[1] "Creating Seurat object..."
[1] "Normalizing Seurat object..."
Performing log-normalization
0% 10 20 30 40 50 60 70 80 90 100%
[----|----|----|----|----|----|----|----|----|----|
**|
[1] "Finding variable genes..."
Calculating gene variances
0% 10 20 30 40 50 60 70 80 90 100%
[----|----|----|----|----|----|----|----|----|----|
**|
Calculating feature variances of standardized and clipped values
0% 10 20 30 40 50 60 70 80 90 100%
[----|----|----|----|----|----|----|----|----|----|
**|
[1] "Scaling data..."
Centering and scaling data matrix
|======================================================================| 100%
[1] "Running PCA..."
[1] "Calculating PC distance matrix..."
[1] "Computing pANN..."
[1] "Classifying doublets.."
Warning messages:
1: package ‘fields’ was built under R version 4.2.3
2: package ‘spam’ was built under R version 4.2.3
3: package ‘viridisLite’ was built under R version 4.2.3
4: package ‘KernSmooth’ was built under R version 4.2.3
Error in collect():
! Failed to collect lazy table.
Caused by error in db_collect():
! Arguments in ... must be used.
✖ Problematic argument:
• ..1 = Inf
ℹ Did you misspell an argument name?
Backtrace:
▆
- ├─get(opt$reference)()
- │ └─celldex:::.create_se(...)
- │ └─celldex:::.ExperimentHub()
- │ └─ExperimentHub::ExperimentHub()
- │ └─AnnotationHub::.Hub(...)
- │ └─AnnotationHub:::.create_cache(...)
- │ └─BiocFileCache::BiocFileCache(cache = cache, ask = ask)
- │ └─BiocFileCache:::.sql_create_db(bfc)
- │ └─BiocFileCache:::.sql_validate_version(bfc)
- │ └─BiocFileCache:::.sql_schema_version(bfc)
- │ ├─base::tryCatch(...)
- │ │ └─base (local) tryCatchList(expr, classes, parentenv, handlers)
- │ └─tbl(src, "metadata") %>% collect(Inf)
- ├─dplyr::collect(., Inf)
- └─dbplyr:::collect.tbl_sql(., Inf)
- ├─base::tryCatch(...)
- │ └─base (local) tryCatchList(expr, classes, parentenv, handlers)
- │ └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
- │ └─base (local) doTryCatch(return(expr), name, parentenv, handler)
- └─dbplyr::db_collect(x$src$con, sql, n = n, warn_incomplete = warn_incomplete, ...)
- └─rlang (local)
<fn>() - └─rlang:::check_dots(env, error, action, call)
- └─rlang:::action_dots(...)
- ├─base (local) try_dots(...)
- └─rlang (local) action(...)
Execution halted
[Sat Nov 4 11:36:51 2023]
Error in rule normalization_pca_rds:
jobid: 2
input: analyses/raw/percent_mt~10/resolution~0.8/10X_17_028.rds
output: analyses/processed/percent_mt~10/resolution~0.8/10X_17_028.rds
shell:
workflow/scripts/scrna-normalization-pca.R --rds analyses/raw/percent_mt~10/resolution~0.8/10X_17_028.rds --doublet.filter --normalization.method LogNormalize --cpu 5 --scale.factor 10000 --reference BlueprintEncodeData --variable.selection.method vst --nfeature 2000 --resolution 0.8 --output.rds analyses/processed/percent_mt~10/resolution~0.8/10X_17_028.rds --umap --tsne
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2023-11-04T113522.930249.snakemake.log (scrna-workflow) unam@IP:~/scrna-workflow$
sinanugur
Hi, thanks so much for making this program it looks really promising. I had a couple of issues I was hoping to get help with.
My specifications are: OS is ubuntu 20.04, linux 5.4 conda version 23.9.0 mamba version 1.5.3 python version 3.10.13.final.0
I downloaded cellsnake with no issues installed the R packages and got the 'all packages are OK' message and ran cellsnake standard data on the practice fetal-brain dataset provided in the workflow tutorial. This is the path to the data files (cellsnake) unam@IP:~/Downloads/data$ ls 10X_17_028 10X_17_029
When I run 'cellsnake standard data' (cellsnake) unam@IP:~/Downloads$ cellsnake standard data
It runs and I get some of the output files but then it terminates due to this following error. Any suggestions on how I can resolve this? Thanks!
Execution halted [Fri Nov 3 15:52:34 2023] Error in rule normalization_pca_rds: jobid: 9 input: analyses/raw/percent_mt~10/resolution~0.8/10X_17_028.rds output: analyses/processed/percent_mt~10/resolution~0.8/10X_17_028.rds shell: /home/unam@IP/miniforge3/envs/cellsnake/lib/python3.9/site-packages/cellsnake/scrna/workflow/scripts/scrna-normalization-pca.R --rds analyses/raw/percent_mt~10/resolution~0.8/10X_17_028.rds --doublet.filter --normalization.method LogNormalize --cpu 1 --scale.factor 10000 --reference BlueprintEncodeData --variable.selection.method vst --nfeature 2000 --resolution 0.8 --output.rds analyses/processed/percent_mt~10/resolution~0.8/10X_17_028.rds --umap --tsne
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2023-11-03T153327.979328.snakemake.log Traceback (most recent call last): File "/home/#####/#####/miniforge3/envs/cellsnake/bin/cellsnake", line 10, in
sys.exit(main())
File "/home//#####/#####//miniforge3/envs/cellsnake/lib/python3.9/site-packages/cellsnake/command_line.py", line 379, in main
run_workflow(cli_arguments)
File "/home//#####/#####//miniforge3/envs/cellsnake/lib/python3.9/site-packages/cellsnake/command_line.py", line 351, in run_workflow
subprocess.check_call(str(snakemake_argument),shell=True)
File "/home//#####/#####//miniforge3/envs/cellsnake/lib/python3.9/subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'snakemake --retries 5 --rerun-incomplete -k -j 1 -s /home//#####/#####//miniforge3/envs/cellsnake/lib/python3.9/site-packages/cellsnake/scrna/workflow/Snakefile --config datafolder=data cellsnake_path=/home//#####/#####//miniforge3/envs/cellsnake/lib/python3.9/site-packages/cellsnake/scrna/ show_labels=True marker_plots_per_cluster_n=20 min_percentage_to_plot=5 reduction=cca highly_variable_features=2000 umap_markers_plot=True test_use=wilcox resolution=0.8 singler_ref=BlueprintEncodeData celltypist_model=Immune_All_Low.pkl mapping=org.Hs.eg.db min_features=200 percent_mt=10 max_molecules=Inf min_molecules=0 normalization_method=LogNormalize scale_factor=10000 metadata_column=condition confidence=0.05 organism=hsa percent_rp=0 variable_selection_method=vst doublet_filter=True species=human microbiome_min_features=3 metadata=None microbiome_min_cells=1 max_features=Inf logfc_threshold=0.25 tsne_markers_plot=False min_cells=3 min_hit_groups=4 taxa=genus dims=30 runid=haz29875 option=standard' returned non-zero exit status 1
I then attempted to integrate the data to see if it would run but I get the following output:
(cellsnake) unam:IP:~/Downloads$ cellsnake integrate data {'10X_17_029', '10X_17_028'} Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 1 (use --cores to define parallelism) Rules claiming more threads will be scaled down. Job stats: job count
all 1 total 1
Select jobs to execute...
[Fri Nov 3 15:53:54 2023] localrule all: jobid: 0 reason: Rules with neither input nor output files are always executed. resources: tmpdir=/tmp