The following error was reported when I ran CNN models with 4 gpus under bonito 0.4.0, but it works fine on bonito 0.2.3. I'd appreciate any suggestions!
[loading data]
[validation set not found: splitting training set]
[loading model]
[0/1672047]: 0%| | [00:00]
Traceback (most recent call last):
File "/mnt/gvol/anaconda3/envs/bonito/bin/bonito", line 33, in
sys.exit(load_entry_point('ont-bonito', 'console_scripts', 'bonito')())
File "/mnt/gvol/bonito/bonito/init.py", line 34, in main
args.func(args)
File "/mnt/gvol/bonito/bonito/cli/train.py", line 104, in main
train_loss, duration = train(
File "/mnt/gvol/bonito/bonito/training.py", line 154, in train
losses = criterion(scores, targets.to(device), lengths.to(device))
File "/mnt/gvol/bonito/bonito/training.py", line 121, in ctc_label_smoothing_loss
loss = ctc_loss(log_probs.to(torch.float32), targets, log_probs_lengths, lengths, reduction='mean')
File "/mnt/gvol/gzy/anaconda3/envs/bonito/lib/python3.8/site-packages/torch/nn/functional.py", line 2304, in ctc_loss
return torch.ctc_loss(
RuntimeError: target_lengths must be of size batch_size
The following error was reported when I ran CNN models with 4 gpus under bonito 0.4.0, but it works fine on bonito 0.2.3. I'd appreciate any suggestions!
[loading data] [validation set not found: splitting training set] [loading model] [0/1672047]: 0%| | [00:00] Traceback (most recent call last): File "/mnt/gvol/anaconda3/envs/bonito/bin/bonito", line 33, in
sys.exit(load_entry_point('ont-bonito', 'console_scripts', 'bonito')())
File "/mnt/gvol/bonito/bonito/init.py", line 34, in main
args.func(args)
File "/mnt/gvol/bonito/bonito/cli/train.py", line 104, in main
train_loss, duration = train(
File "/mnt/gvol/bonito/bonito/training.py", line 154, in train
losses = criterion(scores, targets.to(device), lengths.to(device))
File "/mnt/gvol/bonito/bonito/training.py", line 121, in ctc_label_smoothing_loss
loss = ctc_loss(log_probs.to(torch.float32), targets, log_probs_lengths, lengths, reduction='mean')
File "/mnt/gvol/gzy/anaconda3/envs/bonito/lib/python3.8/site-packages/torch/nn/functional.py", line 2304, in ctc_loss
return torch.ctc_loss(
RuntimeError: target_lengths must be of size batch_size
Thank you in advance!