augmenter object in neuralnets retraining takes 40 secs

vibbic / biosegment

BioSegment is a software stack to enable segmentation of microscopy data using machine learning models.

https://vibbic.github.io/biosegment/

GNU General Public License v3.0

3 stars 2 forks source link

augmenter object in neuralnets retraining takes 40 secs #29

Open berombau opened 4 years ago

berombau commented 4 years ago

l 167 in gpu_worker/app/worker.py takes 40 secs to run

JorisRoels commented 4 years ago

This has to do with the RandomDeformation_2D augmenter that has to build several deformation grids before training (happens here).

Building a deformation grid takes a significant amount of CPU time, which can make GPU computations inefficient during training. Therefore, the augmenter builds the grids in advance and saves them to the GPU so that the augmentation during training happens on the GPU.