Hello! I'm using a dataset with 2 classes with tiny yolo. As long as I leave the configuration intact (i.e. 80 classes, 255 filters) everything trains fine, except evaluation crashes (because 2 classes). However, when I change the number of classes to 2 and filters to 21, the loss becomes NaN. Actually, loss is NaN even if the # classes is 79 and filters is 252. Only time training works is for 80 classes and 255 filters.
Note: The second-layer output is NaN in these cases, causing the loss to be NaN. If anyone knows how to print a hidden layer's output during training, that would be very helpful with debugging.
Has anyone run into this issue? Has anyone even been able to use a custom dataset?
Training images are 512x512 if that may make any difference.
Hello! I'm using a dataset with 2 classes with tiny yolo. As long as I leave the configuration intact (i.e. 80 classes, 255 filters) everything trains fine, except evaluation crashes (because 2 classes). However, when I change the number of classes to 2 and filters to 21, the loss becomes NaN. Actually, loss is NaN even if the # classes is 79 and filters is 252. Only time training works is for 80 classes and 255 filters.
Note: The second-layer output is NaN in these cases, causing the loss to be NaN. If anyone knows how to print a hidden layer's output during training, that would be very helpful with debugging.
Has anyone run into this issue? Has anyone even been able to use a custom dataset?
Training images are 512x512 if that may make any difference.