Is this what the training should look like ?

[shahaamirbader]

Hi I have been training the model for two days now on 2 x V100 GPUs 32 GB. This is what the output at my terminal looks like. Can you comment on if the training is going in the right direction? I see some losses increasing here. Also how many epochs is the training suppose to last for?

[ZikangZhou]

The training looks good. In your screenshot, isn't the loss generally decreasing (some losses are negative)? You can monitor the training process using tensorboard or wandb.

The default max_epoch is set to 64, but you can adjust those training hyperparameters, as it seems that your batch size is currently set to 8.

[HIT-five]

How long will the model training last if I use the 8 NVIDIA 3090? thank you~

[ZikangZhou]

How long will the model training last if I use the 8 NVIDIA 3090? thank you~

Two days I think.