We get no scalability of Hydronet. Regardless whether we use GPUs=1 or GPUs=2. We always process 836 batches. Although the batch size can be different. The validity loss is a big number and it is concerning:
GPU=1, batch 836, time: 17.65s | valid loss 197770579.8095
GPU=2
| end of epoch 0 | time: 26.50s | valid loss 788673.9881 |
| end of epoch 0 | time: 26.50s | valid loss 788673.9929 |
We get no scalability of Hydronet. Regardless whether we use GPUs=1 or GPUs=2. We always process 836 batches. Although the batch size can be different. The validity loss is a big number and it is concerning:
GPU=1, batch 836, time: 17.65s | valid loss 197770579.8095
GPU=2
| end of epoch 0 | time: 26.50s | valid loss 788673.9881 |
| end of epoch 0 | time: 26.50s | valid loss 788673.9929 |