Open tracelail opened 8 months ago
WackerO
commented
7 months ago Hey there, thanks for the bug report! Without the dataset, I won't be able to look into this. Maybe someone else stumbles across this error as well and can share their data for reproducing :)
tracelail
commented
2 months ago Archive.zip Hello,
Hope you are doing well! I have created some reproducible data to elicit the error: "Error in cor(): supply both 'x' and 'y' or a matrix-like 'x'". Attached you will find the *example-params.json*, *example-samplesheet.csv*, *example-contrasts.csv*, and *example-matrix.tsv*.
The attached data will produce the error below when running the nf-core/differentialabundance pipeline:
<details><summary>"nextflow run" output </summary>
~/local-diffabundance ▓▒░ make nextflow-differentialabundance-example ░▒▓ ✔ base 09:20:43 nextflow run nf-core/differentialabundance -r 1.5.0 -profile podman -params-file example-params.json
N E X T F L O W ~ version 24.04.3
Launching https://github.com/nf-core/differentialabundance [redacted_run] DSL2 - revision: 3dd360fed0 [1.5.0]
,--./,-.
___ __ __ __ ___ /,-._.--~'/ \ |__) |__ } { | \| | \__, \__/ | \ |___ \-.,--, .,._,'
nf-core/differentialabundance v1.5.0-g3dd360fCore Nextflow options revision : 1.5.0 runName : redacted_run containerEngine : podman container : [RMARKDOWNNOTEBOOK:biocontainers/r-shinyngs:1.8.8--r43hdfd78af_0] launchDir : /Users/user/local-diffabundance workDir : /Users/user/local-diffabundance/work projectDir : /Users/user/.nextflow/assets/nf-core/differentialabundance userName : user profile : podman configFiles :
Input/output options study_name : redacted_study_name input : s3://company-workflow-results/analyses/CG-2024-775/2024-07-18--CG-2024-775_example-samplesheet.csv contrasts : s3://company-workflow-results/analyses/CG-2024-775/2024-07-18--CG-2024-775__example-contrasts.csv outdir : /Users/user/local-diffabundance/outdir-example
Abundance values matrix : s3://company-workflow-results/analyses/CG-2024-775/2024-07-18--CG-2024-775_rsem.merged.example-gene_counts.tsv
Filtering filtering_min_abundance : 1 filtering_min_samples : 1
Exploratory analysis exploratory_log2_assays : raw,normalised
Differential analysis differential_min_fold_change: 2 differential_max_pval : 1
Limma specific options (microarray only) limma_p_value : 1
gprofiler2 gprofiler2_correction_method: gSCS gprofiler2_background_file : auto
Shiny app settings shinyngs_shinyapps_account : company-rnaseq shinyngs_shinyapps_app_name : redacted_study_name
Reporting options email : user.user@company.com
Max job request options max_cpus : 7 max_memory : 16.GB
If you use nf-core/differentialabundance for your analysis please cite:
The pipeline https://doi.org/10.5281/zenodo.7568000
The nf-core framework https://doi.org/10.1038/s41587-020-0439-x
WARN: Access to undefined parameter gtf -- Initialise it to a default value eg. params.gtf = some_value
executor > local (44)
[3a/4ab044] NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:VALIDATOR (2024-07-18--CG-2024-775_example-samplesheet.csv) [100%] 1 of 1 ✔
[cc/ed5dee] NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:CUSTOM_MATRIXFILTER ([id:redacted_study_name]) [100%] 1 of 1 ✔
[d9/8e4162] NFC…-timepoint1h, blocking:, exclude_samples_col:group_treatment2_vs_treatment1_timepoint1h_exclude, exclude_samples_values:True]) [100%] 1 of 1 ✔
[ed/9b34c0] NFC…replicate;bio_replicate, exclude_samples_col:group_treatment2_vs_treatment1_timepoint3h_exclude, exclude_samples_values:True]) [100%] 12 of 12 ✔
[b5/f07c30] NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:FILTER_DIFFTABLE (12) [100%] 12 of 12 ✔
[3e/4e7f12] NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:PLOT_EXPLORATORY (group_treatment2_vs_treatment1_timepoint1h) [100%] 3 of 3 ✔
[97/a86960] NFC…ALABUNDANCE:DIFFERENTIALABUNDANCE:PLOT_DIFFERENTIAL (group_treatment2_vs_treatment1_timepoint3h_block_biorep_block_techbiorep) [100%] 12 of 12 ✔
[55/fdb625] NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:SHINYNGS_APP (redacted_study_name) [100%] 1 of 1 ✔
[24/2e6497] NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:RMARKDOWNNOTEBOOK (redacted_study_name) [ 0%] 0 of 1
[- ] NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:MAKE_REPORT_BUNDLE -
ERROR ~ Error executing process > 'NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:RMARKDOWNNOTEBOOK (redacted_study_name)'
Caused by:
Process NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:RMARKDOWNNOTEBOOK (redacted_study_name) terminated with an error exit status (1)
Command executed:
cat <<"END_PARAMS_SECTION" > ./.params.yml cpus: 2 artifact_dir: artifacts input_dir: ./ meta: id: redacted_study_name study_name: redacted_study_name study_type: rnaseq study_abundance_type: counts report_file: !!org.codehaus.groovy.runtime.GStringImpl {} report_title: 'null' report_author: 'null' report_contributors: null report_description: 'null' report_scree: true report_round_digits: 4 observations_type: sample observations_id_col: sample observations_name_col: null features: matrix_as_anno.feature_metadata.tsv features_type: gene features_id_col: gene_id features_name_col: gene_name features_metadata_cols: gene_id,gene_name,gene_biotype features_gtf_feature_type: transcript features_gtf_table_first_field: gene_id filtering_min_samples: 1 filtering_min_abundance: 1 filtering_min_proportion: null filtering_grouping_var: null filtering_min_proportion_not_na: 0.5 filtering_min_samples_not_na: null exploratory_main_variable: auto_pca exploratory_clustering_method: ward.D2 exploratory_cor_method: spearman exploratory_n_features: 500 exploratory_whisker_distance: 1.5 exploratory_mad_threshold: -5 exploratory_assay_names: raw,normalised,variance_stabilised exploratory_final_assay: variance_stabilised exploratory_log2_assays: raw,normalised exploratory_palette_name: Set1 differential_file_suffix: null differential_feature_id_column: gene_id differential_feature_name_column: gene_name differential_fc_column: log2FoldChange differential_pval_column: pvalue differential_qval_column: padj differential_min_fold_change: 2 differential_max_pval: 1 differential_max_qval: 0.05 differential_foldchanges_logged: true differential_palette_name: Set1 differential_subset_to_contrast_samples: false deseq2_test: Wald deseq2_fit_type: parametric deseq2_sf_type: ratio deseq2_min_replicates_for_replace: 7 deseq2_use_t: false deseq2_lfc_threshold: 0 deseq2_alt_hypothesis: greaterAbs deseq2_independent_filtering: true deseq2_p_adjust_method: BH deseq2_alpha: 0.1 deseq2_minmu: 0.5 deseq2_vs_method: vst deseq2_shrink_lfc: true deseq2_cores: 1 deseq2_vs_blind: true deseq2_vst_nsub: 1000 gene_sets_files: 'null' observations: 2024-07-18--CG-2024-775_example-samplesheet.sample_metadata.tsv raw_matrix: 2024-07-18--CG-2024-775_rsem.merged.example-gene_counts.assay.tsv normalised_matrix: all.normalised_counts.tsv variance_stabilised_matrix: all.vst.tsv contrasts_file: 2024-07-18--CG-2024-775__example-contrasts.contrasts_file.tsv versions_file: collated_versions.yml logo: nf-core-differentialabundance_logo_light.png css: nf-core_style.css citations: CITATIONS.md END_PARAMS_SECTION
mkdir artifacts
export MKL_NUM_THREADS="2" export OPENBLAS_NUM_THREADS="2" export OMP_NUM_THREADS="2"
mv "differentialabundance_report.Rmd" "differentialabundance_report.Rmd.orig" cp -L "differentialabundance_report.Rmd.orig" "redacted_study_name.Rmd"
Rscript - <<EOF params = yaml::read_yaml('.params.yml')
# Instead of rendering with params, produce a version of the R
# markdown with param definitions set, so the notebook itself can
# be reused
rmd_content <- readLines('redacted_study_name.Rmd')
# Extract YAML content between the first two '---'
start_idx <- which(rmd_content == "---")[1]
end_idx <- which(rmd_content == "---")[2]
rmd_yaml_content <- paste(rmd_content[(start_idx+1):(end_idx-1)], collapse = "\n")
rmd_params <- yaml::yaml.load(rmd_yaml_content)
# Override the params
rmd_params[['params']] <- modifyList(rmd_params[['params']], params)
# Recursive function to add 'value' to list elements, except for top-level
add_value_recursively <- function(lst, is_top_level = FALSE) {
if (!is.list(lst)) {
return(lst)
}
lst <- lapply(lst, add_value_recursively)
if (!is_top_level) {
lst <- list(value = lst)
}
return(lst)
}
# Reformat nested lists under 'params' to have a 'value' key recursively
rmd_params[['params']] <- add_value_recursively(rmd_params[['params']], is_top_level = TRUE)
# Convert back to YAML string
updated_yaml_content <- as.character(yaml::as.yaml(rmd_params))
# Remove the old YAML content
rmd_content <- rmd_content[-((start_idx+1):(end_idx-1))]
# Insert the updated YAML content at the right position
rmd_content <- append(rmd_content, values = unlist(strsplit(updated_yaml_content, split = "\n")), after = start_idx)
writeLines(rmd_content, 'redacted_study_name.parameterised.Rmd')
# Render based on the updated file
rmarkdown::render('redacted_study_name.parameterised.Rmd', output_file='redacted_study_name.html', envir = new.env())
writeLines(capture.output(sessionInfo()), "session_info.log")EOF
cat <<-END_VERSIONS > versions.yml "NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:RMARKDOWNNOTEBOOK": rmarkdown: $(Rscript -e "cat(paste(packageVersion('rmarkdown'), collapse='.'))") END_VERSIONS
Command exit status: 1
Command output: 1/72 2/72 [unnamed-chunk-1] 3/72 4/72 [unnamed-chunk-2] 5/72 6/72 [unnamed-chunk-3] 7/72 8/72 [unnamed-chunk-4] 9/72 10/72 [setup] 11/72 12/72 [unnamed-chunk-5] 13/72 14/72 [unnamed-chunk-6] 15/72 16/72 [unnamed-chunk-7] 17/72 18/72 [unnamed-chunk-8] 19/72 20/72 [unnamed-chunk-9] 21/72 22/72 [unnamed-chunk-10] 23/72 24/72 [unnamed-chunk-11] 25/72 26/72 [unnamed-chunk-12] 27/72 28/72 [unnamed-chunk-13] 29/72 30/72 [unnamed-chunk-14] 31/72 32/72 [unnamed-chunk-15] 33/72 34/72 [unnamed-chunk-16] 35/72 36/72 [unnamed-chunk-17] 37/72 38/72 [unnamed-chunk-18] 39/72 40/72 [unnamed-chunk-19] 41/72 42/72 [unnamed-chunk-20] 43/72 44/72 [unnamed-chunk-21]
Command error: 8/72 [unnamed-chunk-4] 9/72 10/72 [setup] 11/72 12/72 [unnamed-chunk-5] 13/72 14/72 [unnamed-chunk-6] 15/72 16/72 [unnamed-chunk-7] executor > local (44) [3a/4ab044] NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:VALIDATOR (2024-07-18--CG-2024-775_example-samplesheet.csv) [100%] 1 of 1 ✔ [cc/ed5dee] NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:CUSTOM_MATRIXFILTER ([id:redacted_study_name]) [100%] 1 of 1 ✔ [d9/8e4162] NFC…-timepoint1h, blocking:, exclude_samples_col:group_treatment2_vs_treatment1_timepoint1h_exclude, exclude_samples_values:True]) [100%] 1 of 1 ✔ [ed/9b34c0] NFC…replicate;bio_replicate, exclude_samples_col:group_treatment2_vs_treatment1_timepoint3h_exclude, exclude_samples_values:True]) [100%] 12 of 12 ✔ [b5/f07c30] NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:FILTER_DIFFTABLE (12) [100%] 12 of 12 ✔ [3e/4e7f12] NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:PLOT_EXPLORATORY (group_treatment2_vs_treatment1_timepoint1h) [100%] 3 of 3 ✔ [97/a86960] NFC…ALABUNDANCE:DIFFERENTIALABUNDANCE:PLOT_DIFFERENTIAL (group_treatment2_vs_treatment1_timepoint3h_block_biorep_block_techbiorep) [100%] 12 of 12 ✔ [55/fdb625] NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:SHINYNGS_APP (redacted_study_name) [100%] 1 of 1 ✔ [24/2e6497] NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:RMARKDOWNNOTEBOOK (redacted_study_name) [100%] 1 of 1, failed: 1 ✘ [- ] NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:MAKE_REPORT_BUNDLE - Execution cancelled -- Finishing pending tasks before exit ERROR ~ Error executing process > 'NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:RMARKDOWNNOTEBOOK (redacted_study_name)'
Caused by:
Process NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:RMARKDOWNNOTEBOOK (redacted_study_name) terminated with an error exit status (1)
Command executed:
cat <<"END_PARAMS_SECTION" > ./.params.yml cpus: 2 artifact_dir: artifacts input_dir: ./ meta: id: redacted_study_name study_name: redacted_study_name study_type: rnaseq study_abundance_type: counts report_file: !!org.codehaus.groovy.runtime.GStringImpl {} report_title: 'null' report_author: 'null' report_contributors: null report_description: 'null' report_scree: true report_round_digits: 4 observations_type: sample observations_id_col: sample observations_name_col: null features: matrix_as_anno.feature_metadata.tsv features_type: gene features_id_col: gene_id features_name_col: gene_name features_metadata_cols: gene_id,gene_name,gene_biotype features_gtf_feature_type: transcript features_gtf_table_first_field: gene_id filtering_min_samples: 1 filtering_min_abundance: 1 filtering_min_proportion: null filtering_grouping_var: null filtering_min_proportion_not_na: 0.5 filtering_min_samples_not_na: null exploratory_main_variable: auto_pca exploratory_clustering_method: ward.D2 exploratory_cor_method: spearman exploratory_n_features: 500 exploratory_whisker_distance: 1.5 exploratory_mad_threshold: -5 exploratory_assay_names: raw,normalised,variance_stabilised exploratory_final_assay: variance_stabilised exploratory_log2_assays: raw,normalised exploratory_palette_name: Set1 differential_file_suffix: null differential_feature_id_column: gene_id differential_feature_name_column: gene_name differential_fc_column: log2FoldChange differential_pval_column: pvalue differential_qval_column: padj differential_min_fold_change: 2 differential_max_pval: 1 differential_max_qval: 0.05 differential_foldchanges_logged: true differential_palette_name: Set1 differential_subset_to_contrast_samples: false deseq2_test: Wald deseq2_fit_type: parametric deseq2_sf_type: ratio deseq2_min_replicates_for_replace: 7 deseq2_use_t: false deseq2_lfc_threshold: 0 deseq2_alt_hypothesis: greaterAbs deseq2_independent_filtering: true deseq2_p_adjust_method: BH deseq2_alpha: 0.1 deseq2_minmu: 0.5 deseq2_vs_method: vst deseq2_shrink_lfc: true deseq2_cores: 1 deseq2_vs_blind: true deseq2_vst_nsub: 1000 gene_sets_files: 'null' observations: 2024-07-18--CG-2024-775_example-samplesheet.sample_metadata.tsv raw_matrix: 2024-07-18--CG-2024-775_rsem.merged.example-gene_counts.assay.tsv normalised_matrix: all.normalised_counts.tsv variance_stabilised_matrix: all.vst.tsv contrasts_file: 2024-07-18--CG-2024-775__example-contrasts.contrasts_file.tsv versions_file: collated_versions.yml logo: nf-core-differentialabundance_logo_light.png css: nf-core_style.css citations: CITATIONS.md END_PARAMS_SECTION
mkdir artifacts
export MKL_NUM_THREADS="2" export OPENBLAS_NUM_THREADS="2" export OMP_NUM_THREADS="2"
mv "differentialabundance_report.Rmd" "differentialabundance_report.Rmd.orig" cp -L "differentialabundance_report.Rmd.orig" "redacted_study_name.Rmd"
Rscript - <<EOF params = yaml::read_yaml('.params.yml')
# Instead of rendering with params, produce a version of the R
# markdown with param definitions set, so the notebook itself can
# be reused
rmd_content <- readLines('redacted_study_name.Rmd')
# Extract YAML content between the first two '---'
start_idx <- which(rmd_content == "---")[1]
end_idx <- which(rmd_content == "---")[2]
rmd_yaml_content <- paste(rmd_content[(start_idx+1):(end_idx-1)], collapse = "\n")
rmd_params <- yaml::yaml.load(rmd_yaml_content)
# Override the params
rmd_params[['params']] <- modifyList(rmd_params[['params']], params)
# Recursive function to add 'value' to list elements, except for top-level
add_value_recursively <- function(lst, is_top_level = FALSE) {
if (!is.list(lst)) {
return(lst)
}
lst <- lapply(lst, add_value_recursively)
if (!is_top_level) {
lst <- list(value = lst)
}
return(lst)
}
# Reformat nested lists under 'params' to have a 'value' key recursively
rmd_params[['params']] <- add_value_recursively(rmd_params[['params']], is_top_level = TRUE)
# Convert back to YAML string
updated_yaml_content <- as.character(yaml::as.yaml(rmd_params))
# Remove the old YAML content
rmd_content <- rmd_content[-((start_idx+1):(end_idx-1))]
# Insert the updated YAML content at the right position
rmd_content <- append(rmd_content, values = unlist(strsplit(updated_yaml_content, split = "\n")), after = start_idx)
writeLines(rmd_content, 'redacted_study_name.parameterised.Rmd')
# Render based on the updated file
rmarkdown::render('redacted_study_name.parameterised.Rmd', output_file='redacted_study_name.html', envir = new.env())
writeLines(capture.output(sessionInfo()), "session_info.log")EOF
cat <<-END_VERSIONS > versions.yml "NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:RMARKDOWNNOTEBOOK": rmarkdown: $(Rscript -e "cat(paste(packageVersion('rmarkdown'), collapse='.'))") END_VERSIONS
Command exit status: 1
Command output: 1/72 2/72 [unnamed-chunk-1] 3/72 4/72 [unnamed-chunk-2] 5/72 6/72 [unnamed-chunk-3] 7/72 8/72 [unnamed-chunk-4] 9/72 10/72 [setup] 11/72 12/72 [unnamed-chunk-5] 13/72 14/72 [unnamed-chunk-6] 15/72 16/72 [unnamed-chunk-7] 17/72 18/72 [unnamed-chunk-8] 19/72 20/72 [unnamed-chunk-9] 21/72 22/72 [unnamed-chunk-10] 23/72 24/72 [unnamed-chunk-11] 25/72 26/72 [unnamed-chunk-12] 27/72 28/72 [unnamed-chunk-13] 29/72 30/72 [unnamed-chunk-14] 31/72 32/72 [unnamed-chunk-15] 33/72 34/72 [unnamed-chunk-16] 35/72 36/72 [unnamed-chunk-17] 37/72 38/72 [unnamed-chunk-18] 39/72 40/72 [unnamed-chunk-19] 41/72 42/72 [unnamed-chunk-20] 43/72 44/72 [unnamed-chunk-21]
Command error: 8/72 [unnamed-chunk-4] 9/72 10/72 [setup] 11/72 12/72 [unnamed-chunk-5] 13/72 14/72 [unnamed-chunk-6] 15/72 16/72 [unnamed-chunk-7] executor > local (44) [3a/4ab044] NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:VALIDATOR (2024-07-18--CG-2024-775_example-samplesheet.csv) [100%] 1 of 1 ✔ [cc/ed5dee] NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:CUSTOM_MATRIXFILTER ([id:redacted_study_name]) [100%] 1 of 1 ✔ [d9/8e4162] NFC…-timepoint1h, blocking:, exclude_samples_col:group_treatment2_vs_treatment1_timepoint1h_exclude, exclude_samples_values:True]) [100%] 1 of 1 ✔ [ed/9b34c0] NFC…replicate;bio_replicate, exclude_samples_col:group_treatment2_vs_treatment1_timepoint3h_exclude, exclude_samples_values:True]) [100%] 12 of 12 ✔ [b5/f07c30] NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:FILTER_DIFFTABLE (12) [100%] 12 of 12 ✔ [3e/4e7f12] NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:PLOT_EXPLORATORY (group_treatment2_vs_treatment1_timepoint1h) [100%] 3 of 3 ✔ [97/a86960] NFC…ALABUNDANCE:DIFFERENTIALABUNDANCE:PLOT_DIFFERENTIAL (group_treatment2_vs_treatment1_timepoint3h_block_biorep_block_techbiorep) [100%] 12 of 12 ✔ [55/fdb625] NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:SHINYNGS_APP (redacted_study_name) [100%] 1 of 1 ✔ [24/2e6497] NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:RMARKDOWNNOTEBOOK (redacted_study_name) [100%] 1 of 1, failed: 1 ✘ [- ] NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:MAKE_REPORT_BUNDLE - Execution cancelled -- Finishing pending tasks before exit ERROR ~ Error executing process > 'NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:RMARKDOWNNOTEBOOK (redacted_study_name)'
Caused by:
Process NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:RMARKDOWNNOTEBOOK (redacted_study_name) terminated with an error exit status (1)
Command executed:
cat <<"END_PARAMS_SECTION" > ./.params.yml cpus: 2 artifact_dir: artifacts input_dir: ./ meta: id: redacted_study_name study_name: redacted_study_name study_type: rnaseq study_abundance_type: counts report_file: !!org.codehaus.groovy.runtime.GStringImpl {} report_title: 'null' report_author: 'null' report_contributors: null report_description: 'null' report_scree: true report_round_digits: 4 observations_type: sample observations_id_col: sample observations_name_col: null features: matrix_as_anno.feature_metadata.tsv features_type: gene features_id_col: gene_id features_name_col: gene_name features_metadata_cols: gene_id,gene_name,gene_biotype features_gtf_feature_type: transcript features_gtf_table_first_field: gene_id filtering_min_samples: 1 filtering_min_abundance: 1 filtering_min_proportion: null filtering_grouping_var: null filtering_min_proportion_not_na: 0.5 filtering_min_samples_not_na: null exploratory_main_variable: auto_pca exploratory_clustering_method: ward.D2 exploratory_cor_method: spearman exploratory_n_features: 500 exploratory_whisker_distance: 1.5 exploratory_mad_threshold: -5 exploratory_assay_names: raw,normalised,variance_stabilised exploratory_final_assay: variance_stabilised exploratory_log2_assays: raw,normalised exploratory_palette_name: Set1 differential_file_suffix: null differential_feature_id_column: gene_id differential_feature_name_column: gene_name differential_fc_column: log2FoldChange differential_pval_column: pvalue differential_qval_column: padj differential_min_fold_change: 2 differential_max_pval: 1 differential_max_qval: 0.05 differential_foldchanges_logged: true differential_palette_name: Set1 differential_subset_to_contrast_samples: false deseq2_test: Wald deseq2_fit_type: parametric deseq2_sf_type: ratio deseq2_min_replicates_for_replace: 7 deseq2_use_t: false deseq2_lfc_threshold: 0 deseq2_alt_hypothesis: greaterAbs deseq2_independent_filtering: true deseq2_p_adjust_method: BH deseq2_alpha: 0.1 deseq2_minmu: 0.5 deseq2_vs_method: vst deseq2_shrink_lfc: true deseq2_cores: 1 deseq2_vs_blind: true deseq2_vst_nsub: 1000 gene_sets_files: 'null' observations: 2024-07-18--CG-2024-775_example-samplesheet.sample_metadata.tsv raw_matrix: 2024-07-18--CG-2024-775_rsem.merged.example-gene_counts.assay.tsv normalised_matrix: all.normalised_counts.tsv variance_stabilised_matrix: all.vst.tsv contrasts_file: 2024-07-18--CG-2024-775__example-contrasts.contrasts_file.tsv versions_file: collated_versions.yml logo: nf-core-differentialabundance_logo_light.png css: nf-core_style.css citations: CITATIONS.md END_PARAMS_SECTION
mkdir artifacts
export MKL_NUM_THREADS="2" export OPENBLAS_NUM_THREADS="2" export OMP_NUM_THREADS="2"
mv "differentialabundance_report.Rmd" "differentialabundance_report.Rmd.orig" cp -L "differentialabundance_report.Rmd.orig" "redacted_study_name.Rmd"
Rscript - <<EOF params = yaml::read_yaml('.params.yml')
# Instead of rendering with params, produce a version of the R
# markdown with param definitions set, so the notebook itself can
# be reused
rmd_content <- readLines('redacted_study_name.Rmd')
# Extract YAML content between the first two '---'
start_idx <- which(rmd_content == "---")[1]
end_idx <- which(rmd_content == "---")[2]
rmd_yaml_content <- paste(rmd_content[(start_idx+1):(end_idx-1)], collapse = "\n")
rmd_params <- yaml::yaml.load(rmd_yaml_content)
# Override the params
rmd_params[['params']] <- modifyList(rmd_params[['params']], params)
# Recursive function to add 'value' to list elements, except for top-level
add_value_recursively <- function(lst, is_top_level = FALSE) {
if (!is.list(lst)) {
return(lst)
}
lst <- lapply(lst, add_value_recursively)
if (!is_top_level) {
lst <- list(value = lst)
}
return(lst)
}
# Reformat nested lists under 'params' to have a 'value' key recursively
rmd_params[['params']] <- add_value_recursively(rmd_params[['params']], is_top_level = TRUE)
# Convert back to YAML string
updated_yaml_content <- as.character(yaml::as.yaml(rmd_params))
# Remove the old YAML content
rmd_content <- rmd_content[-((start_idx+1):(end_idx-1))]
# Insert the updated YAML content at the right position
rmd_content <- append(rmd_content, values = unlist(strsplit(updated_yaml_content, split = "\n")), after = start_idx)
writeLines(rmd_content, 'redacted_study_name.parameterised.Rmd')
# Render based on the updated file
rmarkdown::render('redacted_study_name.parameterised.Rmd', output_file='redacted_study_name.html', envir = new.env())
writeLines(capture.output(sessionInfo()), "session_info.log")EOF
cat <<-END_VERSIONS > versions.yml "NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:RMARKDOWNNOTEBOOK": rmarkdown: $(Rscript -e "cat(paste(packageVersion('rmarkdown'), collapse='.'))") END_VERSIONS
Command exit status: 1
Command output: 1/72 2/72 [unnamed-chunk-1] 3/72 4/72 [unnamed-chunk-2] 5/72 6/72 [unnamed-chunk-3] 7/72 8/72 [unnamed-chunk-4] 9/72 10/72 [setup] 11/72 12/72 [unnamed-chunk-5] 13/72 14/72 [unnamed-chunk-6] 15/72 16/72 [unnamed-chunk-7] 17/72 18/72 [unnamed-chunk-8] 19/72 20/72 [unnamed-chunk-9] 21/72 22/72 [unnamed-chunk-10] 23/72 24/72 [unnamed-chunk-11] 25/72 26/72 [unnamed-chunk-12] 27/72 28/72 [unnamed-chunk-13] 29/72 30/72 [unnamed-chunk-14] 31/72 32/72 [unnamed-chunk-15] 33/72 34/72 [unnamed-chunk-16] 35/72 36/72 [unnamed-chunk-17] 37/72 38/72 [unnamed-chunk-18] 39/72 40/72 [unnamed-chunk-19] 41/72 42/72 [unnamed-chunk-20] 43/72 44/72 [unnamed-chunk-21]
Command error: 8/72 [unnamed-chunk-4] 9/72 10/72 [setup] 11/72 12/72 [unnamed-chunk-5] 13/72 14/72 [unnamed-chunk-6] 15/72 16/72 [unnamed-chunk-7] 17/72 18/72 [unnamed-chunk-8] 19/72 20/72 [unnamed-chunk-9] 21/72 22/72 [unnamed-chunk-10] 23/72 24/72 [unnamed-chunk-11] 25/72 26/72 [unnamed-chunk-12] 27/72 28/72 [unnamed-chunk-13] 29/72 30/72 [unnamed-chunk-14] 31/72 32/72 [unnamed-chunk-15] 33/72 34/72 [unnamed-chunk-16] 35/72 36/72 [unnamed-chunk-17] 37/72 38/72 [unnamed-chunk-18] 39/72 40/72 [unnamed-chunk-19] 41/72 42/72 [unnamed-chunk-20] 43/72 44/72 [unnamed-chunk-21]
Quitting from lines at lines 721-767 [unnamed-chunk-21] (redacted_study_name.parameterised.Rmd)
Error in cor():
! supply both 'x' and 'y' or a matrix-like 'x'
Backtrace:
Work dir: /Users/user/local-diffabundance/work/24/2e6497d6608e3e571c93b8752e50d3
Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named .command.sh
-- Check '.nextflow.log' file for details
</details>I performed some local troubleshooting by removing all blocking and exclusions from the contrast file (i.e. maintaining the four required fields id, variable , reference, target) but the error still persists.
WackerO
commented
2 months ago Hello @tracelail, thanks for the dataset! The error is caused by the NA values in your samplesheet 🙂 I tried running the pipeline again after filling those in with a copy of some other value in the same column, and then the pipeline finished without issues.
Description of the bug
Hello, I am a co-worker of olgabot and this issue is following issue 196. Running the nf-core/differentialabundance pipeline with the current dev version has resolved the
PLOT_EXPLORATORYstep issue but a potentially similar issue arises later on in the pipeline at theMAKE_REPORT_BUNDLEstep. In addition, although theSHINYNGS_APPstep succeeds and publishes a shiny app to shinyapp.io, it has errors and crashes.Command used and terminal output
The exit status of the task that caused the workflow execution to fail was: 1
Error executing process > 'NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:RMARKDOWNNOTEBOOK (REDACTED-study_name)'
Caused by: Essential container in task exited
Command executed:
Dump .params.yml heredoc (section will be empty if parametrization is disabled)
cat <<"END_PARAMS_SECTION" > ./.params.yml cpus: 2 artifact_dir: artifacts input_dir: ./ meta: id: REDACTED-study_name study_name: REDACTED-study_name study_type: rnaseq study_abundance_type: counts report_file: /.nextflow/assets/nf-core/differentialabundance/assets/differentialabundance_report.Rmd report_title: None report_author: None report_description: None report_scree: true report_round_digits: 4 observations_type: sample observations_id_col: sample observations_name_col: sample features: matrix_as_anno.feature_metadata.tsv features_type: gene features_id_col: gene_id features_name_col: gene_name features_metadata_cols: gene_id,gene_name,gene_biotype features_log2_assays: null features_gtf_feature_type: transcript features_gtf_table_first_field: gene_id filtering_min_samples: 1 filtering_min_abundance: 1 filtering_min_proportion: null filtering_grouping_var: null exploratory_main_variable: auto_pca exploratory_clustering_method: ward.D2 exploratory_cor_method: spearman exploratory_n_features: 500 exploratory_whisker_distance: 1.5 exploratory_mad_threshold: -5 exploratory_assay_names: raw,normalised,variance_stabilised exploratory_final_assay: variance_stabilised exploratory_palette_name: Set1 differential_file_suffix: .deseq2.results.tsv differential_feature_id_column: gene_id differential_feature_name_column: gene_name differential_fc_column: log2FoldChange differential_pval_column: pvalue differential_qval_column: padj differential_min_fold_change: 2 differential_max_pval: 1 differential_max_qval: 0.05 differential_foldchanges_logged: true differential_palette_name: Set1 differential_subset_to_contrast_samples: false deseq2_test: Wald deseq2_fit_type: parametric deseq2_sf_type: ratio deseq2_min_replicates_for_replace: 7 deseq2_use_t: false deseq2_lfc_threshold: 0 deseq2_alt_hypothesis: greaterAbs deseq2_independent_filtering: true deseq2_p_adjust_method: BH deseq2_alpha: 0.1 deseq2_minmu: 0.5 deseq2_vs_method: vst deseq2_shrink_lfc: true deseq2_cores: 1 deseq2_vs_blind: true deseq2_vst_nsub: 1000 gene_sets_files: null observations: REDACTED-file.tsv raw_matrix: rsem.merged.gene_counts.assay.tsv normalised_matrix: all.normalised_counts.tsv variance_stabilised_matrix: all.vst.tsv contrasts_file: REDACTED-file.tsv versions_file: software_versions.yml logo: nf-core-differentialabundance_logo_light.png css: nf-core_style.css citations: CITATIONS.md END_PARAMS_SECTION
Create output directory
mkdir artifacts
Set parallelism for BLAS/MKL etc. to avoid over-booking of resources
export MKL_NUM_THREADS="2" export OPENBLAS_NUM_THREADS="2" export OMP_NUM_THREADS="2"
Work around https://github.com/rstudio/rmarkdown/issues/1508
If the symbolic link is not replaced by a physical file
output- and temporary files will be written to the original directory.
mv "differentialabundance_report.Rmd" "differentialabundance_report.Rmd.orig" cp -L "differentialabundance_report.Rmd.orig" "REDACTED-study_name.Rmd"
Render notebook
Rscript - <<EOF params = yaml::read_yaml('.params.yml')
EOF
cat <<-END_VERSIONS > versions.yml "NFCORE_DIFFERENTIALABUNDANCE:DIFFERENTIALABUNDANCE:RMARKDOWNNOTEBOOK": rmarkdown: $(Rscript -e "cat(paste(packageVersion('rmarkdown'), collapse='.'))") END_VERSIONS
Command exit status: 1
Command output: 1/72
2/72 [unnamed-chunk-1] 3/72
4/72 [unnamed-chunk-2] 5/72
6/72 [unnamed-chunk-3] 7/72
8/72 [unnamed-chunk-4] 9/72
10/72 [setup]
11/72
12/72 [unnamed-chunk-5] 13/72
14/72 [unnamed-chunk-6] 15/72
16/72 [unnamed-chunk-7] 17/72
18/72 [unnamed-chunk-8] 19/72
20/72 [unnamed-chunk-9] 21/72
22/72 [unnamed-chunk-10] 23/72
24/72 [unnamed-chunk-11] 25/72
26/72 [unnamed-chunk-12] 27/72
28/72 [unnamed-chunk-13] 29/72
30/72 [unnamed-chunk-14] 31/72
32/72 [unnamed-chunk-15] 33/72
34/72 [unnamed-chunk-16] 35/72
36/72 [unnamed-chunk-17] 37/72
38/72 [unnamed-chunk-18] 39/72
40/72 [unnamed-chunk-19] 41/72
42/72 [unnamed-chunk-20] 43/72
44/72 [unnamed-chunk-21]
Command error: (more omitted..) 7/72
8/72 [unnamed-chunk-4] 9/72
10/72 [setup]
11/72
12/72 [unnamed-chunk-5] 13/72
14/72 [unnamed-chunk-6] 15/72
16/72 [unnamed-chunk-7] 17/72
18/72 [unnamed-chunk-8] 19/72
20/72 [unnamed-chunk-9] 21/72
22/72 [unnamed-chunk-10] 23/72
24/72 [unnamed-chunk-11] 25/72
26/72 [unnamed-chunk-12] 27/72
28/72 [unnamed-chunk-13] 29/72
30/72 [unnamed-chunk-14] 31/72
32/72 [unnamed-chunk-15] 33/72
34/72 [unnamed-chunk-16] 35/72
36/72 [unnamed-chunk-17] 37/72
38/72 [unnamed-chunk-18] 39/72
40/72 [unnamed-chunk-19] 41/72
42/72 [unnamed-chunk-20] 43/72
44/72 [unnamed-chunk-21] Quitting from lines 706-752 [unnamed-chunk-21] (REDACTED-study_name.Rmd) Error in
cor(): ! supply both 'x' and 'y' or a matrix-like 'x' Backtrace:Work dir: REDACTED-s3://path
Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named
.command.sh