search

Nesvilab / IonQuant

A label free quantification tool.
Other
15 stars 8 forks source link

multithreading #2

Closed tobiasko closed 4 years ago

tobiasko commented 4 years ago

Hi IonQuant developers,

looking at top results on a linux machine it seems like IonQuant is running a single thread. Is this true? Would it make sense to start one instance for each file (I am using PASEF data) to get things done in parallel?

Best, Tobi

chhh commented 4 years ago

@tobiasko This heavily depends on your IO setup. HDD or SSD? Even if SSD - how fast they are for random reads (as each instance it will be reading gigabytes from different files on disk)

tobiasko commented 4 years ago

@chhh Hmmm, i think the scratch space is HDD @cpanse is this true?

fcyu commented 4 years ago

Hi @tobiasko ,

IonQuant is multi-thread. But the data loading part (which is also the part takes most of the time) is limited by your hard disk rather than CPU. So, you may see that the CPU is not fully used during loading.

Best,

Fengchao

tobiasko commented 4 years ago

ok. the building index step seems to run for most of the time:

2020-04-08 18:14:00 [INFO] - Loading /export/data01/tobiasko/pipeline_test/./A_2/3-2-2020_11-25-33_autoQC01_463_1_Slot1-54.d...
2020-04-08 18:14:18 [INFO] - 10%
2020-04-08 18:14:34 [INFO] - 20%
2020-04-08 18:14:39 [INFO] - 30%
2020-04-08 18:14:43 [INFO] - 40%
2020-04-08 18:14:49 [INFO] - 50%
2020-04-08 18:14:54 [INFO] - 60%
2020-04-08 18:14:58 [INFO] - 70%
2020-04-08 18:15:01 [INFO] - 80%
2020-04-08 18:15:05 [INFO] - 90%
2020-04-08 18:15:09 [INFO] - 100%
2020-04-08 18:15:09 [INFO] - Building index...
2020-04-08 18:33:03 [INFO] - Loading /export/data01/tobiasko/pipeline_test/./A_2/3-2-2020_11-25-33_autoQC01_463_1_Slot1-54.pepXML...
fcyu commented 4 years ago

Hi @tobiasko ,

It implies that you have a relative small (but enough) memory. When the memory is not sufficient, IonQuant will try to delete some loaded scans from the memory, which takes additional time. But, as you can see from top/htop, the building step is multi thread.

Best,

Fengchao

fcyu commented 4 years ago

Hi @tobiasko ,

There is one thing need to investigate though. Your data loading part is very fast, which means that either you have a super fast disk or your data is small. If it's the second case, the index building part should also be fast unless you have a very small memory. Can you help to check if the index building part is indeed using all the CPU cores? And can you tell me your data size (2h gradient?), memory size, and CPU core number?

Thanks,

Fengchao

tobiasko commented 4 years ago

@fcyu I think the data is located on a HDD. Is there something I can check to get you more details? The data files are rather small (20 min LC gradient time, latest otof control version -> internal compression?). Here is a listing of a *.d folder:

tobiasko@fgcz-r-033:/scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d$ ls -la
total 460892
drwxr-xr-x 4 tobiasko SG_Employees       247 Apr  8 12:46 .
drwxr-xr-x 4 tobiasko SG_Employees      4096 Apr  8 18:14 ..
drwxr-xr-x 2 tobiasko SG_Employees       131 Apr  8 12:46 20200227_autoQC01_471.m
drwxr-xr-x 2 tobiasko SG_Employees        66 Apr  8 12:46 2020-03-03_10-29-30_One-column-separation
-rw-r--r-- 1 tobiasko SG_Employees  22736896 Apr  8 12:46 analysis.tdf
-rw-r--r-- 1 tobiasko SG_Employees 448983040 Apr  8 12:46 analysis.tdf_bin
-rw-r--r-- 1 tobiasko SG_Employees    215040 Apr  8 12:46 chromatography-data-pre.sqlite
-rw-r--r-- 1 tobiasko SG_Employees      3250 Apr  8 12:46 configuration.hss
-rw-r--r-- 1 tobiasko SG_Employees      4970 Apr  8 12:46 SampleInfo.xml

I allocated 32 Gig of memory to IonQuant. Memory info from top:

KiB Mem : 10443641+total, 11530139+free, 11027892 used, 91803488+buff/cache
KiB Swap: 13421772+total, 13421670+free,     1016 used. 10237863+avail Mem

Reagarding CPUs:

tobiasko@fgcz-r-033:/scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                128
On-line CPU(s) list:   0-127
Thread(s) per core:    2
Core(s) per socket:    32
Socket(s):             2
NUMA node(s):          8
Vendor ID:             AuthenticAMD
CPU family:            23
Model:                 1
Model name:            AMD EPYC 7501 32-Core Processor
Stepping:              2
CPU MHz:               1200.000
CPU max MHz:           2000.0000
CPU min MHz:           1200.0000
BogoMIPS:              4000.12
Virtualization:        AMD-V
L1d cache:             32K
L1i cache:             64K
L2 cache:              512K
L3 cache:              8192K

Does this help?

fcyu commented 4 years ago

Hi @tobiasko ,

Try to give 64 GB memory if your computer has so many. Could you please help to check if index building step is using all CPU cores?

Thanks,

Fengchao

tobiasko commented 4 years ago

ok. How can I check this?

fcyu commented 4 years ago

After typing top, you will be able to see a list of process. Then, you can see the CPU usage of IonQuant. And when IonQuant is building index, the CPU usage reflect how many cores index building is using.

You can take a screenshot or copy the text for me.

Thanks,

Fengchao

tobiasko commented 4 years ago

during building index

top - 20:03:41 up 148 days,  3:13, 68 users,  load average: 44.44, 55.87, 54.70
Tasks: 1529 total,   1 running, 1528 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.5 sy, 32.2 ni, 67.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 10443641+total, 93076576 free, 20944620 used, 93034297+buff/cache
KiB Swap: 13421772+total, 13421670+free,     1016 used. 10138695+avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 32810 tobiasko  39  19 89.682g 3.284g  23912 S 100.0  0.3  26:48.38 java
 48293 tobiasko  20   0   56848   5692   3508 R   2.0  0.0   0:00.28 top
tobiasko commented 4 years ago

I executed:

tobiasko@fgcz-r-033:/scratch/tobiasko/pipeline_test$ nice -19 java -Xmx64G -jar /usr/local/nesvilab/IonQuant-1.0.0.jar --multidir /scratch/tobiasko/pipeline_test  --psm /scratch/tobiasko/pipeline_test/A_1/psm.tsv /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d
fcyu commented 4 years ago

Hi @tobiasko ,

I am 99% sure that you are using single thread. That explains why your index building step took so long. Unfortunately, I don't know why you can only use one thread, because from IonQuant point-of-view, it is multithread (you may double check that by running IonQuant on your own desktop/laptop w/wt FragPipe). It may be related to your server's config.

Best,

Fengchao

tobiasko commented 4 years ago

I checked a FragPipe GUI log file that was created on the exact same machine and guess what the index step takes 2-3 min. log-fragpipe-run-at_2020-04-06_11-25-10.log So I would say it's not the machine config, but the way IonQuant is called or the calling environment (if something like this exists for executing java). In this respect there are some differences how FragPipe calls IonQuant:

IonQuant [Work dir: /scratch/tobiasko/fragpipe_test/FragPipe_output]
java -Xmx32G -Dbruker.lib.path=/usr/local/nesvilab/ext/bruker -cp /tmp/fragpipe/batmass-io-1.17.2.jar:/tmp/fragpipe/ionquant-1.0.0.jar ionquant.IonQuant --threads 32 --mztol 10 --imtol 0.05 --rttol 0.4 --minfreq 0.50 --plot 0 --psm /scratch/tobiasko/fragpipe_test/FragPipe_output/expB_4/psm.tsv --psm /scratch/tobiasko/fragpipe_test/FragPipe_output/expB_3/psm.tsv --psm /scratch/tobiasko/fragpipe_test/FragPipe_output/expA_2/psm.tsv --psm /scratch/tobiasko/fragpipe_test/FragPipe_output/expA_1/psm.tsv --multidir /scratch/tobiasko/fragpipe_test/FragPipe_output /scratch/tobiasko/fragpipe_test/3-2-2020_11-25-33_autoQC01_463_1_Slot1-54.d expA_1/3-2-2020_11-25-33_autoQC01_463_1_Slot1-54.pepXML /scratch/tobiasko/fragpipe_test/3-3-2020_09-19-55_autoQC01_470_1_Slot1-54.d expA_2/3-3-2020_09-19-55_autoQC01_470_1_Slot1-54.pepXML /scratch/tobiasko/fragpipe_test/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d expB_3/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML /scratch/tobiasko/fragpipe_test/3-3-2020_11-37-59_autoQC01_472_3_Slot1-54.d expB_4/3-3-2020_11-37-59_autoQC01_472_3_Slot1-54.pepXML
IonQuant version IonQuant-1.0.0
Batmass-IO version 1.17.2
timsdata library version timsdata-2-4-4
(c) University of Michigan
System OS: Linux, Architecture: amd64
Java Info: 1.8.0_242, OpenJDK 64-Bit Server VM, Oracle Corporation
JVM started with 28 GB memory
2020-04-06 11:11:45 [INFO] - Parameters:
2020-04-06 11:11:45 [INFO] - threads = 32
2020-04-06 11:11:45 [INFO] - mztol = 10.0
2020-04-06 11:11:45 [INFO] - rttol = 0.4
2020-04-06 11:11:45 [INFO] - imtol = 0.05
2020-04-06 11:11:45 [INFO] - plot = 0
2020-04-06 11:11:45 [INFO] - psm = /scratch/tobiasko/fragpipe_test/FragPipe_output/expA_1/psm.tsv
2020-04-06 11:11:45 [INFO] - psm = /scratch/tobiasko/fragpipe_test/FragPipe_output/expA_2/psm.tsv
2020-04-06 11:11:45 [INFO] - psm = /scratch/tobiasko/fragpipe_test/FragPipe_output/expB_3/psm.tsv
2020-04-06 11:11:45 [INFO] - psm = /scratch/tobiasko/fragpipe_test/FragPipe_output/expB_4/psm.tsv
2020-04-06 11:11:45 [INFO] - multidir = /scratch/tobiasko/fragpipe_test/FragPipe_output
2020-04-06 11:11:45 [INFO] - minfreq = 0.5
2020-04-06 11:11:45 [INFO] - tp = 0
2020-04-06 11:11:45 [INFO] - /scratch/tobiasko/fragpipe_test/3-2-2020_11-25-33_autoQC01_463_1_Slot1-54.d
2020-04-06 11:11:45 [INFO] - /scratch/tobiasko/fragpipe_test/3-3-2020_09-19-55_autoQC01_470_1_Slot1-54.d
2020-04-06 11:11:45 [INFO] - /scratch/tobiasko/fragpipe_test/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d
2020-04-06 11:11:45 [INFO] - /scratch/tobiasko/fragpipe_test/3-3-2020_11-37-59_autoQC01_472_3_Slot1-54.d
2020-04-06 11:11:45 [INFO] - /export/data01/tobiasko/fragpipe_test/FragPipe_output/expA_1/3-2-2020_11-25-33_autoQC01_463_1_Slot1-54.pepXML
2020-04-06 11:11:45 [INFO] - /export/data01/tobiasko/fragpipe_test/FragPipe_output/expA_2/3-3-2020_09-19-55_autoQC01_470_1_Slot1-54.pepXML
2020-04-06 11:11:45 [INFO] - /export/data01/tobiasko/fragpipe_test/FragPipe_output/expB_3/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML
2020-04-06 11:11:45 [INFO] - /export/data01/tobiasko/fragpipe_test/FragPipe_output/expB_4/3-3-2020_11-37-59_autoQC01_472_3_Slot1-54.pepXML
2020-04-06 11:11:45 [INFO] - Checking /scratch/tobiasko/fragpipe_test/3-2-2020_11-25-33_autoQC01_463_1_Slot1-54.d...
2020-04-06 11:11:45 [INFO] - Checking /scratch/tobiasko/fragpipe_test/3-3-2020_09-19-55_autoQC01_470_1_Slot1-54.d...
2020-04-06 11:11:45 [INFO] - Checking /scratch/tobiasko/fragpipe_test/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d...
2020-04-06 11:11:45 [INFO] - Checking /scratch/tobiasko/fragpipe_test/3-3-2020_11-37-59_autoQC01_472_3_Slot1-54.d...
2020-04-06 11:11:46 [INFO] - Loading /scratch/tobiasko/fragpipe_test/3-2-2020_11-25-33_autoQC01_463_1_Slot1-54.d...
2020-04-06 11:12:01 [INFO] - 10%
2020-04-06 11:12:15 [INFO] - 20%
2020-04-06 11:12:18 [INFO] - 30%
2020-04-06 11:12:21 [INFO] - 40%
2020-04-06 11:12:24 [INFO] - 50%
2020-04-06 11:12:27 [INFO] - 60%
2020-04-06 11:12:30 [INFO] - 70%
2020-04-06 11:12:33 [INFO] - 80%
2020-04-06 11:12:36 [INFO] - 90%
2020-04-06 11:12:39 [INFO] - 100%
2020-04-06 11:12:40 [INFO] - Building index...
2020-04-06 11:14:55 [INFO] - Loading /export/data01/tobiasko/fragpipe_test/FragPipe_output/expA_1/3-2-2020_11-25-33_autoQC01_463_1_Slot1-54.pepXML...

A -cp parameter, this ionquant.IonQuant thing. Not sure what it is...could any of this be relevant?

chhh commented 4 years ago

@tobiasko -cp is not relevant, it just tells the JVM which jar files to load at runtime (libraries) and the ionquant.IonQuant thing tells it which java "class" (file with code) contains the main() entry point.

tobiasko commented 4 years ago

@chhh Hmmm ok, then I am out of ideas why this happens.

fcyu commented 4 years ago

@tobiasko There are two things that you may check/try:

  1. IonQuant prints the parameters at the beginning of the log. There is a parameter named threads which indicates how many threads will be used. From my desktop, it shows 
    2020-04-09 09:38:07 [INFO] - threads = 8
  2. Can you delete nice -19 at the beginning of your command and run it?

    I executed:

    tobiasko@fgcz-r-033:/scratch/tobiasko/pipeline_test$ nice -19 java -Xmx64G -jar /usr/local/nesvilab/IonQuant-1.0.0.jar --multidir /scratch/tobiasko/pipeline_test  --psm /scratch/tobiasko/pipeline_test/A_1/psm.tsv /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d

Thanks,

Fengchao

tobiasko commented 4 years ago

without nice -19

tobiasko@fgcz-r-033:/scratch/tobiasko/pipeline_test$ java -Xmx64G -jar /usr/local/nesvilab/IonQuant-1.0.0.jar --threads 32 --multidir /scratch/tobiasko/pipeline_test  --psm /scratch/tobiasko/pipeline_test
/A_1/psm.tsv /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d
IonQuant version IonQuant-1.0.0
Batmass-IO version 1.17.2
timsdata library version timsdata-2-4-4
(c) University of Michigan
System OS: Linux, Architecture: amd64
Java Info: 1.8.0_242, OpenJDK 64-Bit Server VM, Oracle Corporation
JVM started with 57 GB memory
2020-04-09 17:30:22 [INFO] - Parameters:
2020-04-09 17:30:22 [INFO] - threads = 32
2020-04-09 17:30:22 [INFO] - mztol = 10.0
2020-04-09 17:30:22 [INFO] - rttol = 0.4
2020-04-09 17:30:22 [INFO] - imtol = 0.05
2020-04-09 17:30:22 [INFO] - plot = 0
2020-04-09 17:30:22 [INFO] - psm = /scratch/tobiasko/pipeline_test/A_1/psm.tsv
2020-04-09 17:30:22 [INFO] - multidir = /scratch/tobiasko/pipeline_test
2020-04-09 17:30:22 [INFO] - minfreq = 0.5
2020-04-09 17:30:22 [INFO] - tp = 0
2020-04-09 17:30:22 [INFO] - /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d
2020-04-09 17:30:22 [INFO] - /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML
2020-04-09 17:30:22 [INFO] - Checking /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d...
2020-04-09 17:30:22 [INFO] - Loading /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d...
2020-04-09 17:30:46 [INFO] - 10%
2020-04-09 17:30:57 [INFO] - 20%
2020-04-09 17:31:00 [INFO] - 30%
2020-04-09 17:31:03 [INFO] - 40%
2020-04-09 17:31:05 [INFO] - 50%
2020-04-09 17:31:08 [INFO] - 60%
2020-04-09 17:31:11 [INFO] - 70%
2020-04-09 17:31:13 [INFO] - 80%
2020-04-09 17:31:16 [INFO] - 90%
2020-04-09 17:31:19 [INFO] - 100%
2020-04-09 17:31:19 [INFO] - Building index...
2020-04-09 17:33:44 [INFO] - Loading /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML...

including nice -19

tobiasko@fgcz-r-033:/scratch/tobiasko/pipeline_test$ nice -19 java -Xmx64G -jar /usr/local/nesvilab/IonQuant-1.0.0.jar --threads 32 --multidir /scratch/tobiasko/pipeline_test  --psm /scratch/tobiasko/pipe
line_test/A_1/psm.tsv /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d
IonQuant version IonQuant-1.0.0
Batmass-IO version 1.17.2
timsdata library version timsdata-2-4-4
(c) University of Michigan
System OS: Linux, Architecture: amd64
Java Info: 1.8.0_242, OpenJDK 64-Bit Server VM, Oracle Corporation
JVM started with 57 GB memory
2020-04-09 17:35:44 [INFO] - Parameters:
2020-04-09 17:35:44 [INFO] - threads = 32
2020-04-09 17:35:44 [INFO] - mztol = 10.0
2020-04-09 17:35:44 [INFO] - rttol = 0.4
2020-04-09 17:35:44 [INFO] - imtol = 0.05
2020-04-09 17:35:44 [INFO] - plot = 0
2020-04-09 17:35:44 [INFO] - psm = /scratch/tobiasko/pipeline_test/A_1/psm.tsv
2020-04-09 17:35:44 [INFO] - multidir = /scratch/tobiasko/pipeline_test
2020-04-09 17:35:44 [INFO] - minfreq = 0.5
2020-04-09 17:35:44 [INFO] - tp = 0
2020-04-09 17:35:44 [INFO] - /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d
2020-04-09 17:35:44 [INFO] - /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML
2020-04-09 17:35:44 [INFO] - Checking /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d...
2020-04-09 17:35:44 [INFO] - Loading /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d...
2020-04-09 17:36:11 [INFO] - 10%
2020-04-09 17:36:23 [INFO] - 20%
2020-04-09 17:36:26 [INFO] - 30%
2020-04-09 17:36:29 [INFO] - 40%
2020-04-09 17:36:31 [INFO] - 50%
2020-04-09 17:36:34 [INFO] - 60%
2020-04-09 17:36:37 [INFO] - 70%
2020-04-09 17:36:40 [INFO] - 80%
2020-04-09 17:36:42 [INFO] - 90%
2020-04-09 17:36:45 [INFO] - 100%
2020-04-09 17:36:45 [INFO] - Building index...
2020-04-09 17:39:13 [INFO] - Loading /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML...

no --threads, but nice -19

tobiasko@fgcz-r-033:/scratch/tobiasko/pipeline_test$ nice -19 java -Xmx64G -jar /usr/local/nesvilab/IonQuant-1.0.0.jar --multidir /scratch/tobiasko/pipeline_test  --psm /scratch/tobiasko/pipeline_test/A_1
/psm.tsv /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d
IonQuant version IonQuant-1.0.0
Batmass-IO version 1.17.2
timsdata library version timsdata-2-4-4
(c) University of Michigan
System OS: Linux, Architecture: amd64
Java Info: 1.8.0_242, OpenJDK 64-Bit Server VM, Oracle Corporation
JVM started with 57 GB memory
2020-04-09 17:43:17 [INFO] - Parameters:
2020-04-09 17:43:17 [INFO] - threads = 128
2020-04-09 17:43:17 [INFO] - mztol = 10.0
2020-04-09 17:43:17 [INFO] - rttol = 0.4
2020-04-09 17:43:17 [INFO] - imtol = 0.05
2020-04-09 17:43:17 [INFO] - plot = 0
2020-04-09 17:43:17 [INFO] - psm = /scratch/tobiasko/pipeline_test/A_1/psm.tsv
2020-04-09 17:43:17 [INFO] - multidir = /scratch/tobiasko/pipeline_test
2020-04-09 17:43:17 [INFO] - minfreq = 0.5
2020-04-09 17:43:17 [INFO] - tp = 0
2020-04-09 17:43:17 [INFO] - /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d
2020-04-09 17:43:17 [INFO] - /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML
2020-04-09 17:43:17 [INFO] - Checking /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d...
2020-04-09 17:43:17 [INFO] - Loading /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d...
2020-04-09 17:43:43 [INFO] - 10%
2020-04-09 17:43:57 [INFO] - 20%
2020-04-09 17:43:59 [INFO] - 30%
2020-04-09 17:44:02 [INFO] - 40%
2020-04-09 17:44:05 [INFO] - 50%
2020-04-09 17:44:08 [INFO] - 60%
2020-04-09 17:44:10 [INFO] - 70%
2020-04-09 17:44:13 [INFO] - 80%
2020-04-09 17:44:16 [INFO] - 90%
2020-04-09 17:44:18 [INFO] - 100%
2020-04-09 17:44:19 [INFO] - Building index...
2020-04-09 18:02:37 [INFO] - Loading /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML...

I don't want to try no --threads without nicing...I might just kill our server. My interpretation: Not choosing threads creates this problem!

tobiasko commented 4 years ago

Happy easter and thx for helping!

fcyu commented 4 years ago

@tobiasko That's strange. Without --threads, IonQuant would try to get the number of cores by using a Java function Runtime.getRuntime().availableProcessors(). From my test, it works well in both Windows 10 and Linux. Maybe some config in your server let that function return 1 somehow.

Anyway, I am glad that you finally found the solution. Please feel free to contact us if you have any further questions.

Best,

Fengchao

anesvi commented 4 years ago

Thanks Tobi

Fengchao, maybe a note on the IonQuant website that HP Linux users may need to set threads themselves?

Sent from my iPhone

On Apr 9, 2020, at 12:04 PM, Tobias Kockmann notifications@github.com wrote:

 External Email - Use Caution

without nice -19

tobiasko@fgcz-r-033:/scratch/tobiasko/pipeline_test$ java -Xmx64G -jar /usr/local/nesvilab/IonQuant-1.0.0.jar --threads 32 --multidir /scratch/tobiasko/pipeline_test --psm /scratch/tobiasko/pipeline_test /A_1/psm.tsv /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d IonQuant version IonQuant-1.0.0 Batmass-IO version 1.17.2 timsdata library version timsdata-2-4-4 (c) University of Michigan System OS: Linux, Architecture: amd64 Java Info: 1.8.0_242, OpenJDK 64-Bit Server VM, Oracle Corporation JVM started with 57 GB memory 2020-04-09 17:30:22 [INFO] - Parameters: 2020-04-09 17:30:22 [INFO] - threads = 32 2020-04-09 17:30:22 [INFO] - mztol = 10.0 2020-04-09 17:30:22 [INFO] - rttol = 0.4 2020-04-09 17:30:22 [INFO] - imtol = 0.05 2020-04-09 17:30:22 [INFO] - plot = 0 2020-04-09 17:30:22 [INFO] - psm = /scratch/tobiasko/pipeline_test/A_1/psm.tsv 2020-04-09 17:30:22 [INFO] - multidir = /scratch/tobiasko/pipeline_test 2020-04-09 17:30:22 [INFO] - minfreq = 0.5 2020-04-09 17:30:22 [INFO] - tp = 0 2020-04-09 17:30:22 [INFO] - /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d 2020-04-09 17:30:22 [INFO] - /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML 2020-04-09 17:30:22 [INFO] - Checking /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d... 2020-04-09 17:30:22 [INFO] - Loading /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d... 2020-04-09 17:30:46 [INFO] - 10% 2020-04-09 17:30:57 [INFO] - 20% 2020-04-09 17:31:00 [INFO] - 30% 2020-04-09 17:31:03 [INFO] - 40% 2020-04-09 17:31:05 [INFO] - 50% 2020-04-09 17:31:08 [INFO] - 60% 2020-04-09 17:31:11 [INFO] - 70% 2020-04-09 17:31:13 [INFO] - 80% 2020-04-09 17:31:16 [INFO] - 90% 2020-04-09 17:31:19 [INFO] - 100% 2020-04-09 17:31:19 [INFO] - Building index... 2020-04-09 17:33:44 [INFO] - Loading /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML...

including nice -19

tobiasko@fgcz-r-033:/scratch/tobiasko/pipeline_test$ nice -19 java -Xmx64G -jar /usr/local/nesvilab/IonQuant-1.0.0.jar --threads 32 --multidir /scratch/tobiasko/pipeline_test --psm /scratch/tobiasko/pipe line_test/A_1/psm.tsv /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d IonQuant version IonQuant-1.0.0 Batmass-IO version 1.17.2 timsdata library version timsdata-2-4-4 (c) University of Michigan System OS: Linux, Architecture: amd64 Java Info: 1.8.0_242, OpenJDK 64-Bit Server VM, Oracle Corporation JVM started with 57 GB memory 2020-04-09 17:35:44 [INFO] - Parameters: 2020-04-09 17:35:44 [INFO] - threads = 32 2020-04-09 17:35:44 [INFO] - mztol = 10.0 2020-04-09 17:35:44 [INFO] - rttol = 0.4 2020-04-09 17:35:44 [INFO] - imtol = 0.05 2020-04-09 17:35:44 [INFO] - plot = 0 2020-04-09 17:35:44 [INFO] - psm = /scratch/tobiasko/pipeline_test/A_1/psm.tsv 2020-04-09 17:35:44 [INFO] - multidir = /scratch/tobiasko/pipeline_test 2020-04-09 17:35:44 [INFO] - minfreq = 0.5 2020-04-09 17:35:44 [INFO] - tp = 0 2020-04-09 17:35:44 [INFO] - /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d 2020-04-09 17:35:44 [INFO] - /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML 2020-04-09 17:35:44 [INFO] - Checking /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d... 2020-04-09 17:35:44 [INFO] - Loading /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d... 2020-04-09 17:36:11 [INFO] - 10% 2020-04-09 17:36:23 [INFO] - 20% 2020-04-09 17:36:26 [INFO] - 30% 2020-04-09 17:36:29 [INFO] - 40% 2020-04-09 17:36:31 [INFO] - 50% 2020-04-09 17:36:34 [INFO] - 60% 2020-04-09 17:36:37 [INFO] - 70% 2020-04-09 17:36:40 [INFO] - 80% 2020-04-09 17:36:42 [INFO] - 90% 2020-04-09 17:36:45 [INFO] - 100% 2020-04-09 17:36:45 [INFO] - Building index... 2020-04-09 17:39:13 [INFO] - Loading /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML...

no --threads, but nice -19

tobiasko@fgcz-r-033:/scratch/tobiasko/pipeline_test$ nice -19 java -Xmx64G -jar /usr/local/nesvilab/IonQuant-1.0.0.jar --multidir /scratch/tobiasko/pipeline_test --psm /scratch/tobiasko/pipeline_test/A_1 /psm.tsv /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d IonQuant version IonQuant-1.0.0 Batmass-IO version 1.17.2 timsdata library version timsdata-2-4-4 (c) University of Michigan System OS: Linux, Architecture: amd64 Java Info: 1.8.0_242, OpenJDK 64-Bit Server VM, Oracle Corporation JVM started with 57 GB memory 2020-04-09 17:43:17 [INFO] - Parameters: 2020-04-09 17:43:17 [INFO] - threads = 128 2020-04-09 17:43:17 [INFO] - mztol = 10.0 2020-04-09 17:43:17 [INFO] - rttol = 0.4 2020-04-09 17:43:17 [INFO] - imtol = 0.05 2020-04-09 17:43:17 [INFO] - plot = 0 2020-04-09 17:43:17 [INFO] - psm = /scratch/tobiasko/pipeline_test/A_1/psm.tsv 2020-04-09 17:43:17 [INFO] - multidir = /scratch/tobiasko/pipeline_test 2020-04-09 17:43:17 [INFO] - minfreq = 0.5 2020-04-09 17:43:17 [INFO] - tp = 0 2020-04-09 17:43:17 [INFO] - /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d 2020-04-09 17:43:17 [INFO] - /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML 2020-04-09 17:43:17 [INFO] - Checking /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d... 2020-04-09 17:43:17 [INFO] - Loading /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.d... 2020-04-09 17:43:43 [INFO] - 10% 2020-04-09 17:43:57 [INFO] - 20% 2020-04-09 17:43:59 [INFO] - 30% 2020-04-09 17:44:02 [INFO] - 40% 2020-04-09 17:44:05 [INFO] - 50% 2020-04-09 17:44:08 [INFO] - 60% 2020-04-09 17:44:10 [INFO] - 70% 2020-04-09 17:44:13 [INFO] - 80% 2020-04-09 17:44:16 [INFO] - 90% 2020-04-09 17:44:18 [INFO] - 100% 2020-04-09 17:44:19 [INFO] - Building index... 2020-04-09 18:02:37 [INFO] - Loading /scratch/tobiasko/pipeline_test/A_1/3-3-2020_10-29-26_autoQC01_471_2_Slot1-54.pepXML...

I don't want to try no --threads without nicing...I might just kill our server. My interpretation: Not choosing threads creates this problem!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/Nesvilab/IonQuant/issues/2#issuecomment-611610332, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIIMM63AYF5H4EILA5K7W2DRLXWYXANCNFSM4MEB2LGQ.

Electronic Mail is not secure, may not be read every day, and should not be used for urgent or sensitive issues

fcyu commented 4 years ago

@anesvi Sure, I edited the page

Best,

Fengchao