Encounter a error while executing training process

kuielab / mdx-net

KUIELAB-MDX-Net got the 2nd place on the Leaderboard A and the 3rd place on the Leaderboard B in the MDX-Challenge ISMIR 2021

https://www.aicrowd.com/challenges/music-demixing-challenge-ismir-2021/

MIT License

182 stars 21 forks source link

Encounter a error while executing training process #33

Open WangWilly opened 3 years ago

WangWilly commented 3 years ago

Hi, I followed all you mentioned step. And I executed python run.py experiment=multigpu_vocals model=ConvTDFNet_vocals to start training then encounter following problem. 💔

RuntimeError: Early stopping conditioned on metric `val/sdr` which is not available. Pass in or modify your `EarlyStopping` callback to use any of the following: `train/loss`, `train/loss_step`, `train/loss_epoch`

Appreciate to get your reply. ✨✨

cnh2769 commented 3 years ago

Same issue here..!

ws-choi commented 2 years ago

Hi @WangWilly and @cnh2769, We have also encountered the issue several times and I am not sure what it is honestly.

Hmm can you try the below and share the result with me?

go to https://github.com/kuielab/mdx-net/blob/bbd46dbf2ceb26c3fbfbe412b5159bce2366c9c0/configs/callbacks/default.yaml#L11

and insert check_on_train_epoch_end: False

early_stopping:
 ~~~~
 check_on_train_epoch_end: False