StephenChan
commented
1 year ago Our slowest training times for EfficientNet sources:
- s372: 35k train images * 10 pts = ~350k examples, 7h45m
- s3058: 20k train images * 5 pts = ~100k examples, 4h56m
- s3363: 14k train images * 50 pts = ~700k examples, 4h23m
- s3371: 14k train images * 10 pts = ~140k examples, 4h15m
- s3411: 14k train images * 50 pts = ~700k examples, 3h57m
Our slowest (completed) training times for VGG16 sources, with dates included to show that some have been using CoralNet 1.0 specs:
- s1052: 40k train images * 30 pts = ~1.2m examples, 15h08m, 2020/10
- s295: 57k train images * 10 pts = ~570k examples, 14h54m, 2019/11
- s1374: 16k train images * 30 pts = ~480k examples, 9h35m, 2021/03
- s1656: 16k train images * 50 pts = ~800k examples, 9h14m, 2021/09
- s371: 25k train images * 20 pts = ~500k examples, 7h39m, 2020/04
(EDIT: Multiplied images by points to get number of examples, although that may not be exact since the number of points is just the source's default.)
So, VGG16 sources' trainings took somewhere between a factor of 1 to 2 longer than EfficientNet sources' trainings, given the same number of train images. Overall, both can get slow. We can ballpark both as 1 second per training image.
Additionally, as of https://github.com/beijbom/coralnet/pull/409 (merged into production September 2021) we were able to see trainings that started but never completed (as Classifiers stuck in PENDING status). When I checked last month (December 2022) there were stuck trainings in 3 sources: 1052, 1374, and 1656, all VGG16. I don't know if this means those stuck trainings were going even longer than 15 hours, but it seems to be a problem correlated with long training times.
kriegman
Motivation:
Training a CoralNet classifier on 10,000s of images has a train-time measured in hours. This is true for both the MLP / EfficientNet case and the LR / VGG16 case,
although MLP / EfficientNet is the faster case(Edit: actually it's not faster; see next post).As we get larger sources on CoralNet, or as we start to explore having super-classifiers trained on multiple sources' data, the long train times will become more of a concern. Mostly for usability, but eventually also server load at a certain point.
We could encourage more folks to switch from VGG16 to EfficientNet if we improve train time + allow more frequent training in the latter case. (Since VGG16 is deprecated and seems more of a pain to work with, I assume trying to speed that up is not worth it.)
David said an idea from 2020 was to switch from logistic regression to an MLP when the size is large (don't know exact threshold). I'm not sure if this is the actual logic in train_utils.py et al. at the moment; it looks to me like it just uses MLP if EfficientNet, and LR if VGG16? But I could be wrong. Either way, it looks to be using MLP for large training jobs (assuming EfficientNet), which is the concern this issue is highlighting.
Other notes from David:
Training the MLP just uses sklearn MLPClassifier on CPU. We didn't explore training time for this, and except for learning rate, the parameters are vanilla.
Speeding up MLP training would involve:
Don't need to deliver a research paper on this. Just some systematic experimentation to improve speed while maintaining the same level accuracy and having confidence that it's delivering.