search

Optimization-AI / LibAUC

LibAUC: A Deep Learning Library for X-Risk Optimization
https://libauc.org/
MIT License
273 stars 37 forks source link

Weird Training Loss plot for Training Task #42

Closed Soothysay closed 7 months ago

Soothysay commented 8 months ago

Hi, I am working on an extremely imbalanced binary classification problem where I follow the tutorial on the docs to use AUCMLoss with PESG optimizer instead of binary cross entropy loss and ADAM which I would use if I did not use LibAUC for my task. Surprisingly, I notice that the training loss increases with LibAUC and then drops. Training_Loss_True_LIBAUC However, for Binary Cross-Entropy + ADAM, the train loss variation pattern is as follows: Training_Loss_True_v4

This seems surprising. Did you face any issue of this sort for any task? I think any insight on that might help me dig deep into this issue.

Thanks!

GangLii commented 8 months ago

Thanks for your interest in our library. This curve is a little unnormal. We didn't see it before. It might be caused by the extremely imbalanced dataset as you mentioned. Could you share the size and the imbalance ratio of your dataset? What is the batch size you use? Did you use the DualSampler? If you don't use the DualSampler in the extremely imbalanced dataset, the first steps, like 10 steps, may not contain positive samples.

optmai commented 8 months ago

Hi,

The loss produced from running the library (including Pytorch) is misleading. There are for mini-batch data. It does not say anything about the model quality. It is just a means to monitor the progress.

I would suggest you to plot the AUROC or AP directly. You can also try to use our SOAP for AP maximization.

Please let us know if this help address your issue.