dotnet / machinelearning

ML.NET is an open source and cross-platform machine learning framework for .NET.
https://dot.net/ml
MIT License
8.92k stars 1.86k forks source link

Object detection training using CUDA gives decent results, while using CPU always returns nothing or inferior results #7028

Open OldManUnderTheHill opened 4 months ago

OldManUnderTheHill commented 4 months ago

System Information (please complete the following information):

Describe the bug I have a self created training set where I want to detect small marked passages of text (with a text marker). If I use CUDA for object detection training I get like the following result (using these parameters: --epoch 10 --device gpu0 -b 1 -st 0.3 -it 0.5 --width 600 --height 800)

-> ObjectDetectionMulti 0.6115

If I switch to CPU training while keeping all other parameters the result is always 0.0%. Testing the CPU model with some test images will never return any boxes, while the model from GPU training generates nice hits. As a workaround I temporarily use the GPU model with CPU code.

I know what I try to do is a bit strange, but it seems to work out if the GPU is used for training. A similar thing happens when I use the stop sign tutorial data set. Results when using the CPU for 5 epochs gives approx. 0.53 as result, while using the GPU on the same set with the same parameters 0.63.

I added my dataset for further analysis, all used documents are public available on the internet.

LetterMarkerAnnot.zip