Question about training config.

[1800012973]

Hello, I'm running the code with the default training config on A40 and I find it can roughly run 60000 iters per day. However, the default total iters is 800000, taking too much time to finish training. Is it reasonable to adjust some config parameters? For example:

1. Default config only costs 14G GPU memory for training. To make full use of GPU, can I incearse the batch size and decrease the total iterations?
2. I found some datasets don't have many data (like NMR only has about 1000 batch), but I need to optimize 1240000 iters. Is such long training necessary? Can I decrease the number of iters on these small datasets?

[szymanowiczs]

I'd suggest a couple of things:

1. look into your data loader if it's possibly bottlenecking training - it was not in our case but it is a little dependent on your specific hardware. we were able to squeeze out around 120,000 iterations per day.
2. yup, you can increase batch size and decrease iterations. it's possible that increasing batch size will increase computation time though so it's not a surefire win - worth trying though.
3. You can get decent results with fewer iterations (though potentially not state-of-the-art), performance gets better with longer training so you can just stop when you are satisfied with results.

Hope this helps :)