I am trying to estimate the runtime of the four different steps (1.1, 1.2, 2.1, and 2.2) depending on the defined epochs and iterations per epoch.
I know it depends also on the GPU power, the GPU utilization, etc. but it would be very helpful to get a feeling how much time a step takes (before you start it).
For the train part, you can estimate the duration of each step with the following output.
For example: a epoch takes about 6h:37min = 397 min and we have 30 epochs
-> 397 min/epoch * 30 epochs = 11910 min = 198.5 h = 8.3 days
But what is with the duration of the validation part?
It should be added to the duration of the train part.
-> 8.3 days + validation duration = estimated time
So the questions are:
-> How many validations does each of the four step's go through?
-> How many validations does each epoch of the four step's go through?
It would be very nice to get this information.
Maybe, you have a better idea to estimate the runtime of the four step's.
If so, then let me know.
Hello @chenwydj ,
I am trying to estimate the runtime of the four different steps (1.1, 1.2, 2.1, and 2.2) depending on the defined epochs and iterations per epoch.
I know it depends also on the GPU power, the GPU utilization, etc. but it would be very helpful to get a feeling how much time a step takes (before you start it).
For the train part, you can estimate the duration of each step with the following output.
For example: a epoch takes about 6h:37min = 397 min and we have 30 epochs -> 397 min/epoch * 30 epochs = 11910 min = 198.5 h = 8.3 days
But what is with the duration of the validation part? It should be added to the duration of the train part. -> 8.3 days + validation duration = estimated time
So the questions are: -> How many validations does each of the four step's go through? -> How many validations does each epoch of the four step's go through?
It would be very nice to get this information. Maybe, you have a better idea to estimate the runtime of the four step's. If so, then let me know.
Kind regards