vdemichev
commented
1 year ago Hi Gabriel,
Yes, many users reported poor support of >32 threads, apparently a limitation of C++11 threading mechanism DIA-NN uses. Btw, reducing predicted library size is great for speeding things up, please see the docs section on that.
Best, Vadim
gabrielpan147
Hello,
We are using DiaNN on HPC for processing dia data. The CPU node on cluster contains 2 AMD-7702 CPU with 128 cores in total. However, we found it quite slow (70 mins )comparing to run it on 2 Xeon Silver 4214R, which contains 12 cores in total(80 mins).
The slurm command line we use is attached here:
!/bin/bash
SBATCH -J DIANN_test_2
SBATCH --partition=cpu
SBATCH --output=DIANN_multi.out
SBATCH --mem=300GB
SBATCH -N 1 -c 128 --nodelist=cpu12
source $HOME/.bashrc conda activate diann module load singularity/3.7.1
/beegfs/userhome/sumto/diann-1.8.1 --f test.d --lib report-lib.predicted.speclib --threads 128 --verbose 1 --out report_core.tsv --qvalue 0.01 --matrices --var-mods 3 --var-mod UniMod:35,15.994915,M --var-mod UniMod:1,42.010565,*n --monitor-mod UniMod:1 --mass-acc 20 --mass-acc-ms1 20 --relaxed-prot-inf --smart-profiling --peak-center --no-ifs-removal --no-norm
The running log is here:
DIA-NN 1.8.1 (Data-Independent Acquisition by Neural Networks) Compiled on Apr 15 2022 08:45:18 Current date and time: Tue Mar 7 15:52:17 2023 Logical CPU cores: 128 Thread number set to 128 Output will be filtered at 0.01 FDR Precursor/protein x samples expression level matrices will be saved along with the main report Maximum number of variable modifications set to 3 Modification UniMod:35 with mass delta 15.9949 at M will be considered as variable Modification UniMod:1 with mass delta 42.0106 at *n will be considered as variable Highly heuristic protein grouping will be used, to reduce the number of protein groups obtained; this mode is recommended for benchmarking protein ID numbers; use with caution for anything else When generating a spectral library, in silico predicted spectra will be retained if deemed more reliable than experimental ones Fixed-width center of each elution peak will be used for quantification Interference removal from fragment elution curves disabled Normalisation disabled Mass accuracy will be fixed to 2e-05 (MS2) and 2e-05 (MS1) The following variable modifications will be scored: UniMod:1 Unless the spectral library specified was created by this version of DIA-NN, it's strongly recommended to specify a FASTA database and use the 'Reannotate' function to allow DIA-NN to identify peptides which can originate from the N/C terminus of the protein: otherwise site localisation might not work properly for modifications of the protein N-terminus or for modifications which do not allow enzymatic cleavage after the modified residue
1 files will be processed [0:00] Loading spectral library report-lib.predicted.speclib [0:11] Library annotated with sequence database(s): human_swiss_isoformincluded_230218.fasta [0:11] Gene names missing for some isoforms [0:11] Library contains 20383 proteins, and 20166 genes [0:13] Spectral library loaded: 42365 protein isoforms, 70676 protein groups and 10803026 precursors in 3367255 elution groups. [0:19] Initialising library
[0:35] File #1/1 [0:35] Loading run Feb_BP_depleted_whisper40_DIA_S1-A1_1_1125.d For most diaPASEF datasets it is better to manually fix both the MS1 and MS2 mass accuracies to values in the range 10-15 ppm. [1:33] 5525457 library precursors are potentially detectable [1:34] Processing... [44:42] RT window set to 2.049 [44:42] Ion mobility window set to 0.0419278 [44:42] Peak width: 10.968 [44:42] Scan window radius set to 24 [44:43] Recommended MS1 mass accuracy setting: 13.4386 ppm [69:50] Removing low confidence identifications [69:50] Searching PTM decoys [70:26] Removing interfering precursors [70:31] Training neural networks: 8895 targets, 7574 decoys [70:33] Number of IDs at 0.01 FDR: 3036 [70:33] Calculating protein q-values [70:34] Number of genes identified at 1% FDR: 522 (precursor-level), 403 (protein-level) (inference performed using proteotypic peptides only) [70:34] Quantification [70:39] Quantification information saved to Feb_BP_depleted_whisper40_DIA_S1-A1_1_1125.d.quant.
[70:39] Cross-run analysis [70:39] Reading quantification information: 1 files [70:39] Quantifying peptides [70:41] Assembling protein groups [70:46] Quantifying proteins [70:47] Calculating q-values for protein and gene groups [70:47] Calculating global q-values for protein and gene groups [70:47] Writing report [70:47] Report saved to report.tsv. [70:47] Saving precursor levels matrix [70:47] Precursor levels matrix (1% precursor and protein group FDR) saved to /beegfs/userhome/sumto/test/report.pr_matrix.tsv. [70:47] Saving protein group levels matrix [70:47] Protein group levels matrix (1% precursor FDR and protein group FDR) saved to /beegfs/userhome/sumto/test/report.pg_matrix.tsv. [70:47] Saving gene group levels matrix [70:47] Gene groups levels matrix (1% precursor FDR and protein group FDR) saved to /beegfs/userhome/sumto/test/report.gg_matrix.tsv. [70:47] Saving unique genes levels matrix [70:47] Unique genes levels matrix (1% precursor FDR and protein group FDR) saved to /beegfs/userhome/sumto/test/report.unique_genes_matrix.tsv. [70:48] Stats report saved to /beegfs/userhome/sumto/test/report.stats.tsv [70:48] Log saved to /beegfs/userhome/sumto/test/report.log.txt Finished
We would like to ask if there're any possibility to speed the whole program up?
Best Regards, Gabriel