use test-time augmentation for uncertainty estimates

[kjappelbaum]

is probably also cheaper than the ensembles I've been trying to build

[pschwllr]

It's quite an unexplored technique but worked well for yield prediction.

You should just make sure to also train with data augmentation.

[kjappelbaum]

Yep, doing that and also have some ensembles.

[kjappelbaum]

But there is also almost no literature on gpt3 uncertainty and fine tuning ensembles/uncertainty

[kjappelbaum]

@pschwllr did you investigate it as a function of the number of augmentation rounds? That is, how many rounds did you need for converged uncertainty estimates?

[pschwllr]

We had tried up to 10.

https://chemrxiv.org/engage/chemrxiv/article-details/60c75258702a9b726c18c101

[kjappelbaum]

anecdotal evidence, but also seems to do something reasonable here

will quantify when I'm back in the hotel

[kjappelbaum]

that's a calibration curve using "1-standard deviation" as probability for the "is_correct classification task"

[kjappelbaum]

ok, on regression they do not seem very reliable. Even when using 50 rounds, the Prediction Interval Coverage Probability (PICP) is < 30%

[kjappelbaum]

and no meaningful change in predictive performance.

[kjappelbaum]

I'll run some systematic tests to put into the SI, but I do not think it will be very promising.

[pschwllr]

I'm surprised that there is basically no difference. Usually, data augmentation always helps a bit.

What data augmentation approach did you go for in the end?

[pschwllr]

I had a quick look at the data augmentation. If I'm not mistaken, the canonical version of the molecule is not kept in the data when the augmentation is done. This might make the task harder than it should be.

Ideally, the model would have access to the canonical_smiles and X augmented copies.

Same at test time, I would always include the canonical_smiles.