search

liznerski / fcdd

Repository for the Explainable Deep One-Class Classification paper
MIT License
225 stars 62 forks source link

Draw a roc metric plot from another dataset #24

Closed DaehanKim-Korea closed 2 years ago

DaehanKim-Korea commented 2 years ago

I tried drawing it using "--load snapshot.pt", but it is lower than the performance confirmed during training.

Assuming you have a validation set of the same type other than the dataset used for training and testing. In this case, how can I draw the roc curve using snapshot.pt?

Can you tell me the right way?

liznerski commented 2 years ago

At first glance, this seems to be the right way. Train on your initial dataset A (with train split A1 and test split A2). Load the snapshot, change to dataset B, and let the runner validate on B.

Did you try swapping the validation sets? So training on A1 with the "test split" being B, and then loading the snapshot, changing to A2, and letting the runner validate on A2?

DaehanKim-Korea commented 2 years ago

The process you mentioned seems to include training.

I want to plot on a test dataset without training.

I've made some modifications, but I'm still testing on the training data... What am I doing wrong?

liznerski commented 2 years ago

Well, first of all, since the snapshot remembers the trained epochs, loading it and continuing with another dataset should not retrain it unless you increase the number of epochs. In either case, you can just set the number of epochs to zero to make double sure (here).

The ROC plots should always report the test AUC.