search

raspstephan / nwp-downscale

MIT License
38 stars 9 forks source link

Sample diversity too low after weighted sampling #38

Closed raspstephan closed 3 years ago

raspstephan commented 3 years ago

I just noticed that we have a LOT of repeated samples even in a single batch (see below). This probably means that either something is wrong in the weighted sampling or it is working as intended but the weighting is too aggressive. I mean there isn't a single no-precip sample in this batch which just cannot be... @HirtM This is probably why we have such crazy overfitting. I thought something wasn't right. I will investigate ASAP.

image

raspstephan commented 3 years ago

I changed the sample weighting. I don't quite understand why the problem was so extreme. Now the weights are simply the mean precipitation values clipped between 0.01 and 0.1. This includes a lot more zero and low precipitation fields which I think we want. The changes are now applied in the main branch.

HirtM commented 3 years ago

@raspstephan Thanks for finding this out and fixing it! Did you check whether the new weighting changes the training process in a significant way? We get less overfitting now?

raspstephan commented 3 years ago

@raspstephan Thanks for finding this out and fixing it! Did you check whether the new weighting changes the training process in a significant way? We get less overfitting now?

Definitely less overfitting, but I did not quantify it. I tried quickly whether this improved the GAN training in any significant way but with my previous settings it did not.