search

PengNi / deepsignal-plant

Detecting methylation using signal-level features from Nanopore sequencing reads of plants
GNU General Public License v3.0
54 stars 12 forks source link

`MemoryError` in call_mods #23

Closed WeipengMO closed 2 years ago

WeipengMO commented 2 years ago

Hi Peng,

Thank you for developing such an excellent tool.

I encountered some problems when running deepsignal-plant call_mods . When I set --nproc 36, it worked well. However, when I want to increase nproc , it has reported an error MemoryError. 

CUDA_VISIBLE_DEVICES=0,1 deepsignal_plant call_mods --input_path fast5s/ \
  --model_path /scem/work/mowp/softwares/model.dp2.CNN.arabnrice2-1_120m_R9.4plus_tem.bn13_sn16.both_bilstm.epoch6.ckpt \
  --result_file fast5s.C.call_mods.tsv \
  --corrected_group RawGenomeCorrected_000 \
  --motifs C --nproc 70 --nproc_gpu 2
# ===============================================
## parameters:
input_path:
        fast5s/
f5_batch_size:
        30
model_path:
        /scem/work/mowp/softwares/model.dp2.CNN.arabnrice2-1_120m_R9.4plus_tem.bn13_sn16.both_bilstm.epoch6.ckpt
model_type:
        both_bilstm
seq_len:
        13
signal_len:
        16
layernum1:
        3
layernum2:
        1
class_num:
        2
dropout_rate:
        0
n_vocab:
        16
n_embed:
        4
is_base:
        yes
is_signallen:
        yes
batch_size:
        512
hid_rnn:
        256
result_file:
        fast5s.C.call_mods.tsv
recursively:
        yes
corrected_group:
        RawGenomeCorrected_000
basecall_subgroup:
        BaseCalled_template
is_dna:
        yes
normalize_method:
        mad
motifs:
        C
mod_loc:
        0
region:
        None
positions:
        None
reference_path:
        None
nproc:
        70
nproc_gpu:
        2
# ===============================================
[main] call_mods starts..
cuda availability: True
4000 fast5 files in total..
parse the motifs string..
read_fast5 process-381662 starts
read_fast5 process-381669 starts
read_fast5 process-381665 starts
read_fast5 process-381670 starts
read_fast5 process-381677 starts
read_fast5 process-381672 starts
read_fast5 process-381666 starts
read_fast5 process-381674 starts
read_fast5 process-381671 starts
read_fast5 process-381683 starts
read_fast5 process-381698 starts
read_fast5 process-381685 starts
read_fast5 process-381663 starts
read_fast5 process-381667 starts
read_fast5 process-381673 starts
read_fast5 process-381679 starts
read_fast5 process-381664 starts
read_fast5 process-381668 starts
read_fast5 process-381711 starts
read_fast5 process-381706 starts
read_fast5 process-381692 starts
read_fast5 process-381719 starts
read_fast5 process-381741 starts
read_fast5 process-381744 starts
read_fast5 process-381704 starts
read_fast5 process-381738 starts
read_fast5 process-381755 starts
read_fast5 process-381725 starts
read_fast5 process-381717 starts
read_fast5 process-381730 starts
read_fast5 process-381688 starts
read_fast5 process-381733 starts
read_fast5 process-381739 starts
read_fast5 process-381754 starts
read_fast5 process-381702 starts
read_fast5 process-381773 starts
read_fast5 process-381764 starts
read_fast5 process-381694 starts
read_fast5 process-381810 starts
read_fast5 process-381797 starts
read_fast5 process-381795 starts
read_fast5 process-381753 starts
read_fast5 process-381760 starts
read_fast5 process-381750 starts
read_fast5 process-381783 starts
read_fast5 process-381767 starts
read_fast5 process-381815 starts
read_fast5 process-381792 starts
read_fast5 process-381772 starts
read_fast5 process-381833 starts
call_mods process-381839 starts
read_fast5 process-381779 starts
read_fast5 process-381788 starts
call_mods process-381845 starts
read_fast5 process-381821 starts
read_fast5 process-381722 starts
read_fast5 process-381780 starts
read_fast5 process-381836 starts
read_fast5 process-381804 starts
read_fast5 process-381819 starts
read_fast5 process-381777 starts
read_fast5 process-381801 starts
read_fast5 process-381825 starts
read_fast5 process-381802 starts
read_fast5 process-381800 starts
read_fast5 process-381724 starts
write_process-381849 starts
read_fast5 process-381830 starts
read_fast5 process-381816 starts
read_fast5 process-381771 starts
Process Process-68:
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/site-packages/deepsignal_plant/call_modifications.py", line 220, in _call_mods_q
    features_batch = features_batch_q.get()
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 94, in get
    res = self._recv_bytes()
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/connection.py", line 411, in _recv_bytes
    return self._recv(size)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Process Process-29:
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/site-packages/deepsignal_plant/call_modifications.py", line 324, in _read_features_fast5s_q
    errornum_q.put(error)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 87, in put
    self._start_thread()
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 170, in _start_thread
    self._thread.start()
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/threading.py", line 852, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError
Traceback (most recent call last):
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/scem/work/mowp/anaconda3/envs/deepsignalpenv/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
MemoryError

What parameters can be used to speed up deepsignal-plant? Here's my hardware:

CPU: Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz

Memory:  188G

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-PCIE...  Off  | 00000000:58:00.0 Off |                    0 |
| N/A   33C    P0    25W / 250W |      4MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-PCIE...  Off  | 00000000:D8:00.0 Off |                    0 |
| N/A   30C    P0    25W / 250W |      4MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Thanks, Weipeng

PengNi commented 2 years ago

Hi Weipeng, more --nporcs need more memory in deepsignal-plant. Running deepsignal-plant in 70 processes may exceed the 188G memory. (1) How many processors does your machine have? (cat /proc/cpuinfo can check the number). We don't suggest using a number which is bigger than the total number of processors of the machine. (2) If you set --nproc as 36, then you can set --nproc_gpu for example as 6 for faster detection. Also note that the number of --nproc_gpu shouldn't be too big, as it may exceed the GPU memory. (3) If you really want to use a bigger --nproc, try to set a smaller --f5_batch_size and a smaller --batch_size to see if it works.

Best, Peng

WeipengMO commented 2 years ago

Thank you very much! The machine I used has 72 processors. I will try to set the parameters as you suggest.

cpuinfo.txt

Best, Weipeng