Validation before writing paper

[mesarcik]

Experiments for validation of models before committing to writing.

• [x] Determine which samples from each class we are misclassifying (if it seems unreasonable, then reassess model)
• [x] Obtain an estimate of how long inference takes per spectrogram
• [x] Evaluate OOD results in a more appropriate way
  • I think doing it on a per class level is a little weird
  • I think we should randomly drop a class train the classifier and see its overall performance at detecting anomalies
  • Then we can do the same experiments with multiple classes, so see the overall performance wrt to number of classes dropped.
• [x] Dataset finalisation
  • Electric fence?
  • Third order high noise fused with first order?

Results needed for paper:

The purpose of this paper is to show how we can solve the two problems, these being classification of known failures as well as anomaly detection for unseen anomalies.

1) Comparison of different anomaly detection models on our dataset vs our method.

• Note this might be a little weird as vision-based AD models only evaluate in a single class setting.
• I think the single class setting doesn't really make sense, so what we should compare against is fine-tuned ability to do anomaly detection. In the sense that we use all classes and we get it to predict if its anomalous or not. We could also technically do it without subsampling on all available data. 2) Comparison of classifier with and without SSL corrections that proves that our approach is best
• this will probably in the form of the bar graph done in the presentation where we show that we can get a 3% overall increase 3) Comparing the combined anomaly detection performance of the supervised backbone against the ssl augmentation when some classes are OOD
• I envisage this to be a line chart with combined F-2 anomaly detection performance on the y-axis against number of OOD classes on the x-axis.
  4) How many epochs of pre-training/fine tuning repsectively to obtain best performance? 5) Model ablations:
• If we remove parts of the SSL loss we prescribe how does it effect the overall training objective in terms of anomaly detection.
• Combination function of classification and detection

[mesarcik]

Samples that are misclassified:

• Here I use an imperfectly trained model to validate which samples we are misclassifying.
• I assume that anomalies that are classified as normal are most problematic.
• There are 2 things to note in this plot, that class 1 (high noise element) is predicted to be lightining ( i think the high noise element is class problematic)
• The model predicts 20 of the "normal" samples to be a galactic plane (this is also very possible)
• there are 15 misclassified normal samples as anomalous

Anomalous samples that are calssified as normal

High noise element:

• I geuss in this case it is assumed to be RFI

• Not sure, maybe an anomaly is in the normal set?

Lightning:

Galactic Plance

Source in the side lobes:

[mesarcik]

Timing and inference performance:

• Note all calculations here are done using bfloat16 datatypes and on a Nvidia A100

Timing

• From preliminary results (using some code I found online) of an average of 1000 iterations using a non-optimal batch size, I get the following results:
  • supervised model: 1ms -> 960 spectrograms per second
  • ssl backbone + classification head: 1.2ms -> 820 spectrograms per second
  • So i estimate that the total inference time would be about 2ms per spectrogram or about 400 spectra per second

Performance estimation

• I use the defaults settings from the most starred python repo: https://github.com/Lyken17/pytorch-OpCounter
• I got the following results:
  • supervised model = ~10x10e9 MACs -> 5 GFLOPs
  • ssl backbone = ~0.34x10e9 MACs - > 0.17 GFLOPs
  • classification head = very poor performance for some reason?

[mesarcik]

Anomaly detection performance / OOD detection

• below is the barplot comparing the overall anomaly detection performance in terms of f1 score when removing a given class from the training set
• On average I see a performance increase of 3%, in the best case scenario we see an increase of 7% and in the worst case, there is no performance change
• Also note that there is class imbalance between classes, for example we are only using like 20 samples in the oscillating tile class, this means that the 2% improvement in the OOD detection is actually quite meaningful.

Removing multiple classes simultaneously

• Here we randomly remove multiple classes during training and plot that against the anomaly detection performance.
• In this plot it is clear that the SSL training vastly improves the robustness of anomaly detection

Note:

• i think these results are not valid, I think I use a pretrained backbone for the SSL that has seen all classes, but only fine-tune the classification head. Im not sure if it entirely invalidates these results, but I need to re-run them with a non-finetuned backbone.
• Another problem that i just encountered is that we save each model only if the validation accuracy has increased between epochs, however when the number of OOD classes is very high, then we very quickly get to a near perfect validation accuracy, which means we are saving the first few epochs rather than those which have a decrease in loss after training for longer. To counteract this i have a added an additional if statement to fix this problem.

[mesarcik]

Redo of OOD results:

• not sure what is happening with the first class.
• These models are only trained once, so the results aren't completely realistic (i.e. when 1 class is OOD we randomly drop 1 class of the 8, when we should actually train it multiple times with multiple single classes dropped.
• TL;DR looks promising, a little weird, but not final result.

[mesarcik]

Note Resnet152 doesnt want to train.

[mesarcik]

Results:

Changing backbone for SSL model and comparing the resizing trick:

• Seems like resizing doesnt work (maybe need to train for longer, but seems pretty conclusive across all models)

• Seems like increasing model size increases overfitting?

• All results from now on will not use resizing

Changing backbone SSL vs supervised:

• Here I find that electric fence seems to be difficult for the SSL model to detect it
• Here it seems that resizing doest have a big effect on the model performance
• It seems that the classifier benefits from a bigger model, but the ssl doesnt
• I think ViT and ConvNext have a weird bug, because their performance is really bad.
• overall the best performacne is about 0.9 using SSL and resnet18 and resizing ( but this also could be training noise)

Resizing

No Electric Fence

With Electric fence

No Resizing

No Electric Fence

With Electric fence

[mesarcik]

Seems like increasing the resize crop area improves performance:

[mesarcik]

Multiple runs for resnet18 vs resnet50

• TL;DR smaller models perform better
  • (I think this has more to do with the classifier than it does with the anomaly detector)
  • We could try using a resnet18 classifier and a resnet50 detector to see what happens
• I ran each experiment 3x, for 50 epochs (maybe not enough for SSL convergence)
• both have the same crop and resize ranges

Resnet18

Resnet50