danstowell
commented
3 years ago Using masks (whether binary- or soft-masks) has a strong recent history e.g. in audio source separation by deep learning, because they're stable targets for prediction (no need for the dataset to cover all the exact pixel values etc). I'd suggest that using cloud masks is likely to be a reliable route. If you don't have labelled data for that, you could consider synthetically generating data.
I don't have the experience to know whether the errors induced by existing cloud-segmentation algorithms are anything to worry about.
Your just_cloud_image approach is interesting, but raises lots of questions. Clouds don't really affect pixels in an additive way, more a masking way, so modelling it as a delta seems unlikely to create a stable target for inference. (parkinglot - heavycloud is very different from footballpitch - heavycloud, even if the observed heavycloud is identical in both cases!) Worth a pilot test of some sort, sure.
JackKelly
leonoverweel
maxaragon
tcapelle
simonpf
quentinpaletta
Quentin Paletta et al.'s 2021 "ECLIPSE" paper show that it's a good idea to predict cloud masks, because these are concise representations of the sky (with the caveat that I'm afraid I've only skim-read the paper so far, so I may have misunderstood!).
Also, ideally, we'd probably prefer satellite image sequences of just the clouds, rather than the clouds plus the land. Sometimes optical flow messes up by moving the land around (!), so optical flow would probably perform better on 'pure cloud' images (with the land removed). And, for ML approaches for forecasting future satellite images, it's unfair to expect the ML model to reconstruct images of land as the cloud moves away from land, when that information may be entirely absent from the input image sequence (because the cloud has completely covered the land in the input image sequence).
On the other hand, I'm a little nervous about using binary cloud masks because clouds are so varied, and thin wispy clouds which might not be classified as 'cloud' by a binary mask can have a significant effect on irradiance.
Also, reliable cloud mask labels are hard to come by, I think? (Sure, there are algorithms for segmenting clouds, but none are perfect, right?)
So maybe we can automatically subtract 'cloud' from 'land' something like this: If we had a perfect 'cloud-free' satellite image for every time of day, then we could do a pixel-wise subtraction:
just_cloud_image[t] = cloud_free_image[t] - actual_image[t]. Then we could use thejust_cloud_imagefor our downstream models.The question then becomes: How to generate the set of
cloud_free_image[t]for every time of day, and time of year? Could it be as simple as taking the median pixel value, per pixel, at a given time of day, over the last month or so of imagery? e.g. to get thecloud_free_imagefor 12:00, look at all the images taken at 12:00 over the last month, and take the median pixel value?Or maybe the median is the wrong statistic: Maybe instead we can assume the histogram of pixel values at a given time of day, over the last month, would have two peaks: one corresponding to 'cloud free', and the other corresponding to 'cloudy'. And we want the mode of the 'cloud free' peak, which I guess will always be the less-bright peak?
(also see the twitter discussion about this issue)
TODO: