VIB-PSB / MINI-EX

Motif-Informed Network Inference of cell type-specific gene regulatory networks in plants
Other
15 stars 8 forks source link

job finish without excuting certain processes for clustermap etc. #14

Closed e-tomato closed 9 months ago

e-tomato commented 11 months ago

Hello,

I ran the following script to analyze the MINI-EX job. The job completed without any issues, but certain processes were not executed. I would greatly appreciate it if you could help me identify how I can correct my script. Thank you.

my script:

module load bioinfo/Nextflow/23.04.3
module load containers/singularity/3.9.9
nextflow -C miniex_run_slurm.config run miniex.nf -resume 

or nextflow -C miniex_run_slurm.config run miniex.nf

miniex_run_slurm.config:

` executor { name = 'slurm' queueSize = 5 }

process.container = "vibpsb/mini-ex:latest" singularity { enabled = true cacheDir = "singularity_cache" autoMounts = true }

params { expressionMatrix = "$baseDir/tomato/INPUTS/_matrix.txt" markersOut = "$baseDir/tomato/INPUTS/_allMarkers.txt" cell2clusters = "$baseDir/tomato/INPUTS/_cells2clusters.txt" cluster2ident = "$baseDir/tomato/INPUTS/_identities.txt" TF_list = "$baseDir/tomato/INPUTS/TF_list.txt" //termsOfInterest = "$baseDir/tomato/INPUTS/GOsIwant.txt" termsOfInterest = null

//grnboostOut = "/$baseDir/tomato/OUTPUTS/GRNBoost2_output/*_grnboost2.txt"
grnboostOut = null

doMotifAnalysis = true // set to <false> if no motif mapping data is available [CAUTION: without motif data MINI-EX is less reliable]
featureFile_motifs = "$baseDir/data_sly/sly_2023.1_motifMapping.out.gz"
infoTF = "$baseDir/data_sly/sly_TF2fam2mot.txt"
GOfile = "$baseDir/data_sly/sly_go_gene_plaza_all_filt1k_extended_bp_noBP.txt" //set to <null> if GO data is not available (when doing so, termsOfInterest should also be set to <null>)
alias = "$baseDir/data_sly/sly_gene_aliases.txt"

script_enricher = "$baseDir/bin/enricherv2.4"
script_checkInput = "$baseDir/bin/MINIEX_checkInput.py"
script_grnboost = "$baseDir/bin/MINIEX_grnboostMultiprocess.py"
script_motifs = "$baseDir/bin/MINIEX_filterForMotifs.py"
script_topDEGs = "$baseDir/bin/MINIEX_selectTopDEGs.py"
script_expTFs = "$baseDir/bin/MINIEX_filterForTFExp.py"
script_info = "$baseDir/bin/MINIEX_makeInfoFile.py"
script_clustermap = "$baseDir/bin/MINIEX_clustermap.py"
script_networkCentrality = "$baseDir/bin/MINIEX_network_analysis.py"
script_checkReference = "$baseDir/bin/MINIEX_checkRef.py"
script_filesEnrichment = "$baseDir/bin/MINIEX_makeFilesEnrichment.py"
script_makedfRef = "$baseDir/bin/MINIEX_makeRankingDf_ref.py"
script_makedfStd = "$baseDir/bin/MINIEX_makeRankingDf_std.py"

script_makeborda = "$baseDir/bin/MINIEX_makeBorda.py" script_scoreEdges = "$baseDir/bin/MINIEX_scoreEdges.py" script_heatmapTops = "$baseDir/bin/MINIEX_visual_heatmap_top150.py" script_regmaps = "$baseDir/bin/MINIEX_regmap.py"

tops = "700"
expressionFilter = "10"
motifFilter = "TF-F_motifs" // To use the motifs of the TF family, specify "TF-F_motifs". To only use the motifs known for a TF, specify "TF_motifs".
topRegs = "150"

}

process {

withName: check_input_files {
    clusterOptions = '--mem=10G'
}
withName: get_expressed_genes {
    clusterOptions = '--mem=2G'
}
withName: run_grnboost {
    clusterOptions = '--mem=60G -c 5'
}
withName: unzip_motifMappings {
    executor='local'
}
withName: run_enricher_motifs {
    clusterOptions = '--mem=4G'
}
withName: filter_motifs {
    clusterOptions = '--mem=20G'
}
withName: filter_motifs_dummy {
    executor='local'
}
withName: get_topDEGs {
    clusterOptions = '--mem=10G'
}
withName: run_enricher_cluster {
    clusterOptions = '--mem=4G'
}
withName: filter_expression {
    clusterOptions = '--mem=20G'
}
    withName: make_info_file {
    clusterOptions = '--mem=10G'
}
withName: network_centrality {
    clusterOptions = '--mem=20G'
}
withName: getFiles_enrichment {
    clusterOptions = '--mem=10G'
}
withName: GO_enricher {
    clusterOptions = '--mem=4G'
}
    withName: check_reference {
    clusterOptions = '--mem=10G'
}
withName: ranking_df_ref {
    clusterOptions = '--mem=10G'
}
withName: ranking_df_std {
    clusterOptions = '--mem=10G'
}
withName: makeBorda {
    clusterOptions = '--mem=10G'
}
withName: scoreEdges {
    clusterOptions = '--mem=15G'
}
withName: heatmap_tops {
    clusterOptions = '--mem=20G'
}
withName: clustermap_regs {
    clusterOptions = '--mem=40G'
}
withName: regmaps {
    clusterOptions = '--mem=80G'
}

} `

nf report:

N E X T F L O W ~ version 23.04.3 Launching miniex.nf [pedantic_aryabhata] DSL2 - revision: 7d53c557b4 Motif-Informed Network Inference from gene EXpression v.2.0

Running TF motif enrichment filtering on TF-F_motifs Running single-cell cluster enrichment using the top 700 upregulated genes per cluster Filtering out regulons of single-cell clusters where the TF is expressed in less than 10 % of the cells Plotting expression specificity and DE calls for the top 150 regulons

WARN: The echo directive has been deprecated - use to debug instead [- ] process > check_input_files - [- ] process > run_grnboost - [- ] process > get_expressed_genes - [- ] process > unzip_motifMappings - [- ] process > run_enricher_motifs - [- ] process > filter_motifs - [- ] process > get_topDEGs - [- ] process > run_enricher_cluster -

[- ] process > check_input_files - [- ] process > run_grnboost - [- ] process > get_expressed_genes - [- ] process > unzip_motifMappings - [- ] process > run_enricher_motifs - [- ] process > filter_motifs - [- ] process > get_topDEGs - [- ] process > run_enricher_cluster - [- ] process > filter_expression - [- ] process > make_info_file - [- ] process > clustermap_regs - [- ] process > network_centrality - [- ] process > getFiles_enrichment - [- ] process > GO_enricher - [- ] process > ranking_df_std - [- ] process > makeBorda - [- ] process > scoreEdges - [- ] process > heatmap_tops - [- ] process > regmaps - [DPA3, /work/user/kwang/MINI-EX/tomato/INPUTS/DPA3_matrix.txt]

executor > slurm (3) [- ] process > check_input_files - [e8/2a9b3b] process > run_grnboost (1) [ 0%] 0 of 1 [78/75664b] process > get_expressed_genes (1) [ 0%] 0 of 1 [1f/1fd247] process > unzip_motifMappings [100%] 1 of 1, cached: 1 ✔ [- ] process > run_enricher_motifs - [- ] process > filter_motifs - [54/b9d02c] process > get_topDEGs (1) [ 0%] 0 of 1 [- ] process > run_enricher_cluster - [- ] process > filter_expression - [- ] process > make_info_file - [- ] process > clustermap_regs - [- ] process > network_centrality - [- ] process > getFiles_enrichment - [- ] process > GO_enricher - [- ] process > ranking_df_std - [- ] process > makeBorda - [- ] process > scoreEdges - [- ] process > heatmap_tops - [- ] process > regmaps - [DPA3, /work/user/kwang/MINI-EX/tomato/INPUTS/DPA3_matrix.txt]

executor > slurm (3) [- ] process > check_input_files - [e8/2a9b3b] process > run_grnboost (1) [ 0%] 0 of 1 [78/75664b] process > get_expressed_genes (1) [100%] 1 of 1 ✔ [1f/1fd247] process > unzip_motifMappings [100%] 1 of 1, cached: 1 ✔ [- ] process > run_enricher_motifs - [- ] process > filter_motifs - [54/b9d02c] process > get_topDEGs (1) [100%] 1 of 1 ✔ [- ] process > run_enricher_cluster - [- ] process > filter_expression - [- ] process > make_info_file - [- ] process > clustermap_regs - [- ] process > network_centrality - [- ] process > getFiles_enrichment - [- ] process > GO_enricher - [- ] process > ranking_df_std - [- ] process > makeBorda - [- ] process > scoreEdges - [- ] process > heatmap_tops - [- ] process > regmaps - [DPA3, /work/user/kwang/MINI-EX/tomato/INPUTS/DPA3_matrix.txt]

executor > slurm (4) [- ] process > check_input_files - [e8/2a9b3b] process > run_grnboost (1) [100%] 1 of 1 ✔ [78/75664b] process > get_expressed_genes (1) [100%] 1 of 1 ✔ [1f/1fd247] process > unzip_motifMappings [100%] 1 of 1, cached: 1 ✔ [00/1b3c1b] process > run_enricher_motifs (1) [ 0%] 0 of 1 [- ] process > filter_motifs - [54/b9d02c] process > get_topDEGs (1) [100%] 1 of 1 ✔ [- ] process > run_enricher_cluster - [- ] process > filter_expression - [- ] process > make_info_file - [- ] process > clustermap_regs - [- ] process > network_centrality - [- ] process > getFiles_enrichment - [- ] process > GO_enricher - [- ] process > ranking_df_std - [- ] process > makeBorda - [- ] process > scoreEdges - [- ] process > heatmap_tops - [- ] process > regmaps - [DPA3, /work/user/kwang/MINI-EX/work/e8/2a9b3bd797fdf76811b4ec6eb5721b/DPA3_grnboost2.txt]

executor > slurm (4) [- ] process > check_input_files - [e8/2a9b3b] process > run_grnboost (1) [100%] 1 of 1 ✔ [78/75664b] process > get_expressed_genes (1) [100%] 1 of 1 ✔ [1f/1fd247] process > unzip_motifMappings [100%] 1 of 1, cached: 1 ✔ [00/1b3c1b] process > run_enricher_motifs (1) [ 0%] 0 of 1 [- ] process > filter_motifs - [54/b9d02c] process > get_topDEGs (1) [100%] 1 of 1 ✔ [- ] process > run_enricher_cluster - [- ] process > filter_expression - [- ] process > make_info_file - [- ] process > clustermap_regs - [- ] process > network_centrality - [- ] process > getFiles_enrichment - [- ] process > GO_enricher - [- ] process > ranking_df_std - [- ] process > makeBorda - [- ] process > scoreEdges - [- ] process > heatmap_tops - [- ] process > regmaps - [DPA3, /work/user/kwang/MINI-EX/work/e8/2a9b3bd797fdf76811b4ec6eb5721b/DPA3_grnboost2.txt]

executor > slurm (5) [- ] process > check_input_files - [e8/2a9b3b] process > run_grnboost (1) [100%] 1 of 1 ✔ [78/75664b] process > get_expressed_genes (1) [100%] 1 of 1 ✔ [1f/1fd247] process > unzip_motifMappings [100%] 1 of 1, cached: 1 ✔ [00/1b3c1b] process > run_enricher_motifs (1) [100%] 1 of 1 ✔ [c4/ceedbb] process > filter_motifs (1) [ 0%] 0 of 1 [54/b9d02c] process > get_topDEGs (1) [100%] 1 of 1 ✔ [- ] process > run_enricher_cluster - [- ] process > filter_expression - [- ] process > make_info_file - [- ] process > clustermap_regs - [- ] process > network_centrality - [- ] process > getFiles_enrichment - [- ] process > GO_enricher - [- ] process > ranking_df_std - [- ] process > makeBorda - [- ] process > scoreEdges - [- ] process > heatmap_tops - [- ] process > regmaps -

executor > slurm (5) [- ] process > check_input_files - [e8/2a9b3b] process > run_grnboost (1) [100%] 1 of 1 ✔ [78/75664b] process > get_expressed_genes (1) [100%] 1 of 1 ✔ [1f/1fd247] process > unzip_motifMappings [100%] 1 of 1, cached: 1 ✔ [00/1b3c1b] process > run_enricher_motifs (1) [100%] 1 of 1 ✔ [c4/ceedbb] process > filter_motifs (1) [ 0%] 0 of 1 [54/b9d02c] process > get_topDEGs (1) [100%] 1 of 1 ✔ [- ] process > run_enricher_cluster - [- ] process > filter_expression - [- ] process > make_info_file - [- ] process > clustermap_regs - [- ] process > network_centrality - [- ] process > getFiles_enrichment - [- ] process > GO_enricher - [- ] process > ranking_df_std - [- ] process > makeBorda - [- ] process > scoreEdges - [- ] process > heatmap_tops - [- ] process > regmaps -

executor > slurm (6) [- ] process > check_input_files - [e8/2a9b3b] process > run_grnboost (1) [100%] 1 of 1 ✔ [78/75664b] process > get_expressed_genes (1) [100%] 1 of 1 ✔ [1f/1fd247] process > unzip_motifMappings [100%] 1 of 1, cached: 1 ✔ [00/1b3c1b] process > run_enricher_motifs (1) [100%] 1 of 1 ✔ [c4/ceedbb] process > filter_motifs (1) [100%] 1 of 1 ✔ [54/b9d02c] process > get_topDEGs (1) [100%] 1 of 1 ✔ [64/081566] process > run_enricher_cluster (1) [ 0%] 0 of 1 [- ] process > filter_expression - [- ] process > make_info_file - [- ] process > clustermap_regs - [- ] process > network_centrality - [- ] process > getFiles_enrichment - [- ] process > GO_enricher - [- ] process > ranking_df_std - [- ] process > makeBorda - [- ] process > scoreEdges - [- ] process > heatmap_tops - [- ] process > regmaps -

executor > slurm (6) [- ] process > check_input_files - [e8/2a9b3b] process > run_grnboost (1) [100%] 1 of 1 ✔ [78/75664b] process > get_expressed_genes (1) [100%] 1 of 1 ✔ [1f/1fd247] process > unzip_motifMappings [100%] 1 of 1, cached: 1 ✔ [00/1b3c1b] process > run_enricher_motifs (1) [100%] 1 of 1 ✔ [c4/ceedbb] process > filter_motifs (1) [100%] 1 of 1 ✔ [54/b9d02c] process > get_topDEGs (1) [100%] 1 of 1 ✔ [64/081566] process > run_enricher_cluster (1) [ 0%] 0 of 1 [- ] process > filter_expression - [- ] process > make_info_file - [- ] process > clustermap_regs - [- ] process > network_centrality - [- ] process > getFiles_enrichment - [- ] process > GO_enricher - [- ] process > ranking_df_std - [- ] process > makeBorda - [- ] process > scoreEdges - [- ] process > heatmap_tops - [- ] process > regmaps -

executor > slurm (6) [- ] process > check_input_files - [e8/2a9b3b] process > run_grnboost (1) [100%] 1 of 1 ✔ [78/75664b] process > get_expressed_genes (1) [100%] 1 of 1 ✔ [1f/1fd247] process > unzip_motifMappings [100%] 1 of 1, cached: 1 ✔ [00/1b3c1b] process > run_enricher_motifs (1) [100%] 1 of 1 ✔ [c4/ceedbb] process > filter_motifs (1) [100%] 1 of 1 ✔ [54/b9d02c] process > get_topDEGs (1) [100%] 1 of 1 ✔ [64/081566] process > run_enricher_cluster (1) [100%] 1 of 1 ✔ [- ] process > filter_expression - [- ] process > make_info_file - [- ] process > clustermap_regs - [- ] process > network_centrality - [- ] process > getFiles_enrichment - [- ] process > GO_enricher - [- ] process > ranking_df_std - [- ] process > makeBorda - [- ] process > scoreEdges - [- ] process > heatmap_tops - [- ] process > regmaps -

executor > slurm (6) [- ] process > check_input_files - [e8/2a9b3b] process > run_grnboost (1) [100%] 1 of 1 ✔ [78/75664b] process > get_expressed_genes (1) [100%] 1 of 1 ✔ [1f/1fd247] process > unzip_motifMappings [100%] 1 of 1, cached: 1 ✔ [00/1b3c1b] process > run_enricher_motifs (1) [100%] 1 of 1 ✔ [c4/ceedbb] process > filter_motifs (1) [100%] 1 of 1 ✔ [54/b9d02c] process > get_topDEGs (1) [100%] 1 of 1 ✔ [64/081566] process > run_enricher_cluster (1) [100%] 1 of 1 ✔ [- ] process > filter_expression - [- ] process > make_info_file - [- ] process > clustermap_regs - [- ] process > network_centrality - [- ] process > getFiles_enrichment - [- ] process > GO_enricher - [- ] process > ranking_df_std - [- ] process > makeBorda - [- ] process > scoreEdges - [- ] process > heatmap_tops - [- ] process > regmaps - Done! Completed at: 12-Dec-2023 22:07:57 Duration : 11h 9m 2s CPU hours : 11.1 (0% cached) Succeeded : 6 Cached : 1

jstaut commented 10 months ago

We suspect that one of the input files is either missing or incorrectly typed in the config file (mismatch between the actual file name and the name in the config). Probably the problem is in the name of your cells2cluster file (please check carefully if the name and path of that file is corresponding to the one in the config file). Alternatively, it could also be that the path to the expression matrix is wrong. So please check whether the path to both files is completely correct. FYI: we already have implemented a fix so that incorrect file names will, in the future, throw an error instead of simply finishing the pipeline as you showed. This will be released soon in MINI-EX version 2.2.

If this does not solve the problem, could you please show us the content of your input folder (ls -l /work/user/kwang/MINI-EX/tomato/INPUTS/) so we can have a closer look, and we will get back to you.