search

PaddlePaddle / PaddleSpeech

Easy-to-use Speech Toolkit including Self-Supervised Learning model, SOTA/Streaming ASR with punctuation, Streaming TTS with text frontend, Speaker Verification System, End-to-End Speech Translation and Keyword Spotting. Won NAACL2022 Best Demo Award.
https://paddlespeech.readthedocs.io
Apache License 2.0
11.07k stars 1.85k forks source link

AssertionError: condition input's numel should be 1 #2009

Closed 1105135335 closed 2 years ago

1105135335 commented 2 years ago

Describe the bug /home/aistudio/PaddleSpeech/examples/voxceleb/sv0下执行bash ./run.sh --stage 1 --stop_stage 1

To Reproduce Steps to reproduce the behavior:

  1. Go to '/home/aistudio/PaddleSpeech/examples/voxceleb/sv0'
  2. Run 'bash ./run.sh --stage 1 --stop_stage 1'
  3. See error

Screenshots image

Additional context run.sh ：gpus=0

Additional log

aistudio@jupyter-2318074-4109838:~/PaddleSpeech/examples/voxceleb/sv0$ bash ./run.sh --stage 1 --stop_stage 1
using 1 gpus...
LAUNCH INFO 2022-06-02 18:27:10,087 -----------  Configuration  ----------------------
LAUNCH INFO 2022-06-02 18:27:10,087 devices: None
LAUNCH INFO 2022-06-02 18:27:10,087 elastic_level: -1
LAUNCH INFO 2022-06-02 18:27:10,087 elastic_timeout: 30
LAUNCH INFO 2022-06-02 18:27:10,087 gloo_port: 6767
LAUNCH INFO 2022-06-02 18:27:10,087 host: None
LAUNCH INFO 2022-06-02 18:27:10,087 job_id: default
LAUNCH INFO 2022-06-02 18:27:10,087 legacy: False
LAUNCH INFO 2022-06-02 18:27:10,087 log_dir: log
LAUNCH INFO 2022-06-02 18:27:10,087 log_level: INFO
LAUNCH INFO 2022-06-02 18:27:10,087 master: None
LAUNCH INFO 2022-06-02 18:27:10,087 max_restart: 3
LAUNCH INFO 2022-06-02 18:27:10,087 nnodes: 1
LAUNCH INFO 2022-06-02 18:27:10,087 nproc_per_node: None
LAUNCH INFO 2022-06-02 18:27:10,087 rank: -1
LAUNCH INFO 2022-06-02 18:27:10,087 run_mode: collective
LAUNCH INFO 2022-06-02 18:27:10,087 server_num: None
LAUNCH INFO 2022-06-02 18:27:10,087 servers: 
LAUNCH INFO 2022-06-02 18:27:10,087 trainer_num: None
LAUNCH INFO 2022-06-02 18:27:10,088 trainers: 
LAUNCH INFO 2022-06-02 18:27:10,088 training_script: /home/aistudio/PaddleSpeech/paddlespeech/vector/exps/ecapa_tdnn/train.py
LAUNCH INFO 2022-06-02 18:27:10,088 training_script_args: ['--device', 'gpu', '--checkpoint-dir', 'exp/ecapa-tdnn-vox12-big/', '--data-dir', 'data/', '--config', 'conf/ecapa_tdnn.yaml']
LAUNCH INFO 2022-06-02 18:27:10,088 with_gloo: 0
LAUNCH INFO 2022-06-02 18:27:10,088 --------------------------------------------------
LAUNCH WARNING 2022-06-02 18:27:10,088 Compatible mode enable with args ['--gpus=0']
-----------  Configuration Arguments -----------
backend: auto
cluster_topo_path: None
elastic_pre_hook: None
elastic_server: None
enable_auto_mapping: False
force: False
gpus: 0
heter_devices: 
heter_worker_num: None
heter_workers: 
host: None
http_port: None
ips: 127.0.0.1
job_id: None
log_dir: log
np: None
nproc_per_node: None
rank_mapping_path: None
run_mode: None
scale: 0
server_num: None
servers: 
training_script: /home/aistudio/PaddleSpeech/paddlespeech/vector/exps/ecapa_tdnn/train.py
training_script_args: ['--device', 'gpu', '--checkpoint-dir', 'exp/ecapa-tdnn-vox12-big/', '--data-dir', 'data/', '--config', 'conf/ecapa_tdnn.yaml']
worker_num: None
workers: 
------------------------------------------------
WARNING 2022-06-02 18:27:10,089 launch.py:519] Not found distinct arguments and compiled with cuda or xpu or npu or mlu. Default use collective mode
WARNING 2022-06-02 18:27:10,089 launch.py:519] Not found distinct arguments and compiled with cuda or xpu or npu or mlu. Default use collective mode
launch train in GPU mode!
INFO 2022-06-02 18:27:10,089 launch_utils.py:679] Change selected_gpus into reletive values. --ips:0 will change into relative_ips:[0] according to your CUDA_VISIBLE_DEVICES:['0']
INFO 2022-06-02 18:27:10,089 launch_utils.py:679] Change selected_gpus into reletive values. --ips:0 will change into relative_ips:[0] according to your CUDA_VISIBLE_DEVICES:['0']
INFO 2022-06-02 18:27:10,091 launch_utils.py:561] Local start 1 processes. First process distributed environment info (Only For Debug): 
    +=======================================================================================+
    |                        Distributed Envs                      Value                    |
    +---------------------------------------------------------------------------------------+
    |                       PADDLE_TRAINER_ID                        0                      |
    |                 PADDLE_CURRENT_ENDPOINT                 127.0.0.1:52939               |
    |                     PADDLE_TRAINERS_NUM                        1                      |
    |                PADDLE_TRAINER_ENDPOINTS                 127.0.0.1:52939               |
    |                     PADDLE_RANK_IN_NODE                        0                      |
    |                 PADDLE_LOCAL_DEVICE_IDS                        0                      |
    |                 PADDLE_WORLD_DEVICE_IDS                        0                      |
    |                     FLAGS_selected_gpus                        0                      |
    |             FLAGS_selected_accelerators                        0                      |
    +=======================================================================================+

INFO 2022-06-02 18:27:10,091 launch_utils.py:561] Local start 1 processes. First process distributed environment info (Only For Debug): 
    +=======================================================================================+
    |                        Distributed Envs                      Value                    |
    +---------------------------------------------------------------------------------------+
    |                       PADDLE_TRAINER_ID                        0                      |
    |                 PADDLE_CURRENT_ENDPOINT                 127.0.0.1:52939               |
    |                     PADDLE_TRAINERS_NUM                        1                      |
    |                PADDLE_TRAINER_ENDPOINTS                 127.0.0.1:52939               |
    |                     PADDLE_RANK_IN_NODE                        0                      |
    |                 PADDLE_LOCAL_DEVICE_IDS                        0                      |
    |                 PADDLE_WORLD_DEVICE_IDS                        0                      |
    |                     FLAGS_selected_gpus                        0                      |
    |             FLAGS_selected_accelerators                        0                      |
    +=======================================================================================+

INFO 2022-06-02 18:27:10,092 launch_utils.py:566] details about PADDLE_TRAINER_ENDPOINTS can be found in log/endpoints.log, and detail running logs maybe found in log/workerlog.0
INFO 2022-06-02 18:27:10,092 launch_utils.py:566] details about PADDLE_TRAINER_ENDPOINTS can be found in log/endpoints.log, and detail running logs maybe found in log/workerlog.0
launch proc_id:27947 idx:0
/home/aistudio/PaddleSpeech/paddlespeech/vector/modules/loss.py:246: DeprecationWarning: invalid escape sequence \l
  """
---------------------------- augment: True
batch_size: 32
chunk_duration: 3.0
cohort_size: 20000
embedding_mean_norm: True
embedding_std_norm: False
epochs: 10
global_embedding_norm: True
hop_size: 160
learning_rate: 1e-08
log_interval: 10
margin: 0.2
max_lr: 0.001
model:
  attention_channels: 128
  channels: [1024, 1024, 1024, 1024, 3072]
  dilations: [1, 2, 3, 4, 1]
  input_size: 80
  kernel_sizes: [5, 3, 3, 3, 1]
  lin_neurons: 192
n_mels: 80
n_train_snts: 400000
num_speakers: 7205
num_workers: 2
random_chunk: True
save_interval: 10
scale: 30
score_norm: s-norm
seed: 1986
shuffle: True
skip_prep: False
split_ratio: 0.9
sr: 16000
step_size: 140000
verification_file: data/vox1/veri_test2.txt
window_size: 400
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/distributed/parallel.py:158: UserWarning: Currently not a parallel execution environment, `paddle.distributed.init_parallel_env` will not do anything.
  "Currently not a parallel execution environment, `paddle.distributed.init_parallel_env` will not do anything."
2022-06-02 18:27:12.034 | INFO     | paddlespeech.vector.training.seeding:seed_everything:28 - Set the seed of paddle, random, np.random to 1986.
2022-06-02 18:27:12.037 | INFO     | paddlespeech.vector.io.augment:build_augment_pipeline:842 - start to build the augment pipeline
2022-06-02 18:27:12.055 | WARNING  | paddlespeech.vector.io.dataset:load_speaker_to_label:129 - No speaker id to label file
2022-06-02 18:27:12.057 | WARNING  | paddlespeech.vector.io.dataset:load_speaker_to_label:129 - No speaker id to label file
W0602 18:27:12.062904 27947 gpu_context.cc:278] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 10.1
W0602 18:27:12.067212 27947 gpu_context.cc:306] device: 0, cuDNN Version: 7.6.
Traceback (most recent call last):
  File "/home/aistudio/PaddleSpeech/paddlespeech/vector/exps/ecapa_tdnn/train.py", line 366, in <module>
    main(args, config)
  File "/home/aistudio/PaddleSpeech/paddlespeech/vector/exps/ecapa_tdnn/train.py", line 175, in main
    waveforms = waveform_augment(waveforms, augment_pipeline)
  File "/home/aistudio/PaddleSpeech/paddlespeech/vector/io/augment.py", line 895, in waveform_augment
    waveforms_aug = aug(waveforms)  # (N, L)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 930, in __call__
    return self._dygraph_call_func(*inputs, **kwargs)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
    outputs = self.forward(*inputs, **kwargs)
  File "/home/aistudio/PaddleSpeech/paddlespeech/vector/io/augment.py", line 769, in forward
    waveforms = self.speed_perturb(waveforms)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 930, in __call__
    return self._dygraph_call_func(*inputs, **kwargs)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
    outputs = self.forward(*inputs, **kwargs)
  File "/home/aistudio/PaddleSpeech/paddlespeech/vector/io/augment.py", line 419, in forward
    perturbed_waveform = self.resamplers[self.samp_index](waveform)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 930, in __call__
    return self._dygraph_call_func(*inputs, **kwargs)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
    outputs = self.forward(*inputs, **kwargs)
  File "/home/aistudio/PaddleSpeech/paddlespeech/vector/io/augment.py", line 226, in forward
    self._indices_and_weights(waveforms)
  File "/home/aistudio/PaddleSpeech/paddlespeech/vector/io/augment.py", line 367, in _indices_and_weights
    delta_t.masked_select(inside_window_indices)))
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/varbase_patch_methods.py", line 778, in __setitem__
    return _setitem_impl_(self, item, value)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/variable_index.py", line 610, in _setitem_impl_
    return set_value_for_bool_tensor(var, slice_item, value)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/variable_index.py", line 739, in set_value_for_bool_tensor
    cond(item.any(), lambda: idx_not_empty(var, item, value))
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/layers/control_flow.py", line 2451, in cond
    assert pred.size == 1, "condition input's numel should be 1"
AssertionError: condition input's numel should be 1
INFO 2022-06-02 18:27:23,132 launch_utils.py:343] terminate all the procs
INFO 2022-06-02 18:27:23,132 launch_utils.py:343] terminate all the procs
ERROR 2022-06-02 18:27:23,133 launch_utils.py:642] ABORT!!! Out of all 1 trainers, the trainer process with rank=[0] was aborted. Please check its log.
ERROR 2022-06-02 18:27:23,133 launch_utils.py:642] ABORT!!! Out of all 1 trainers, the trainer process with rank=[0] was aborted. Please check its log.
INFO 2022-06-02 18:27:27,137 launch_utils.py:343] terminate all the procs
INFO 2022-06-02 18:27:27,137 launch_utils.py:343] terminate all the procs
INFO 2022-06-02 18:27:27,137 launch.py:402] Local processes completed.
INFO 2022-06-02 18:27:27,137 launch.py:402] Local processes completed.
yt605155624 commented 2 years ago

删掉 paddlespeech/s2t/init.py 里对 size() 函数的重写

1105135335 commented 2 years ago

好的，感谢感谢