10/30/24 One Class Classifier Results

[bgorman3]

Accuracy: 0.5000 Precision: 0.5000 Recall: 1.0000 F1 Score: 0.6667 [[ 0 136] # [TN FP] [ 0 136]] # [FN TP]

Conclusion the model was overfitted. Data Used was 136 Monet from the 10% of the original 1366 and 136 Non-Monet Paintings

Validation Data With predictions

https://docs.google.com/document/d/19_xuKkNB3kwV6Eixrr00069JLbIfGlzwuzJCqmf238o/edit?usp=sharing

[bgorman3]

I'm not entirely too sure how to fix this problem in the future I am using the Default ImageNet Model in line 6 of model.py model = models.resnet101(weights=models.ResNet101_Weights.DEFAULT)

[bgorman3]

Code Used data_preparation.py model.py test_indicies.json config.py f_score.py modelTrainingOneClass.py

[bgorman3]

accuracy = (TP + TN) / (TP + FP + TN + FN) formula

[bgorman3]

Combined Test Set Performance: Accuracy: 0.6385 Precision: 0.6385 Recall: 1.0000
F1 Score: 0.7794

Confusion Matrix: [[ 0 77] [ 0 136]]

I ran the test with my ai_data set that included 77 images and it still classified each one as a Monet FP

[btdobbs]

Thanks for the hard work and sharing these results. Which python file is the main entry for these results?

[bgorman3]

F_score.py was my testing program ModeltrainingOneClass.py was my training program

[btdobbs]

Regarding our discussion last Tuesday, I think we are on the same page regarding the data split for training. You are properly segmenting the Monet and non-Monet paintings. There are 1366 Monet paintings. How many non-Monet paintings are you using? There are obviously less than 1366 paintings in the non-Monet folder, but GitHub doesn't give me a count. I would recommend balancing the paintings by randomly adding 1366 non Monet paintings from a pool of paintings (I have all ~46k WikiArt paintings on SanDisk if you want me to bring that in). You could also try to randomly down sample the Monet paintings.

You also may need to work with the variable prob_monet and prob_non_monet on line 76/77 of f_score.py to adjust your predict threshold. It looks like it is set to 100% and 0% and I am not sure if the code you are using returns a binary response.

As a side note, the Monet accuracy in the WikiArt 90 class experiment from my first paper is .8251. (This link is the spreadsheet of all metrics). I would think your accuracy should fall close to that if not higher.

[btdobbs]

You also want to make sure that the paintings used in the test code (i.e. f_score.py data/monet_paintings and data/non-monet_paintings) are not used in the training (i.e. data/large_monet_dataset, data/non-monet_test)

[bgorman3]

For my initial experiment, I used the non_monet_test136 folder because it matched the 136 Monet images I was testing. I know I have a lot of datasets, which can be confusing—sorry about that! Now, I need to locate my dataset with 1,366 images to match my Monet set for the new model. I will be using the same split for my non-monet so I need to set up the correct splits for it as well

[btdobbs]

No worries! You are doing great! Let's discuss progress more during Tuesday's meeting, but if you have any questions, please comment on an existing issue or create an issue and assign to me :)