Try The Perceiver

[JackKelly]

"Perceiver: General Perception with Iterative Attention"

https://arxiv.org/abs/2103.03206

https://www.youtube.com/watch?v=P_xeshTnPZg

Principled way to combine multiple "modalities" (which, for us, means "modalities" like satellite imagery, PV timeseries, air quality measurements etc).

directly outputs of the order of 1,000 values. Could be PV yield for different locations (maybe we can enter our own queries to ask for PV yield for specific locations etc) Could pre-train by outputting small patch of CM-SAF, PV yield, and irradiance.

Could be used both where we have predictions of future satellite images (using optical flow or SatFlow); and where we just use 'real' images of the recent history. Can also be used to infer PV yield from a single timestep of satellite data.

Could also 'label' different modalities using an encoding, similar to position encoding. Or maybe a 1-hot encoding; or binary.

Could provide satellite data at multiple spatial resolutions naturally. Eg a vector at full resolution for 128x128 km, then concat a vector of lower res data for a larger field of view, without duplication.

Could maybe even provide a way to combine data from satellites in low Earth orbit with geo stationary data

[JackKelly]

Implementation here: https://github.com/lucidrains/perceiver-pytorch

[JackKelly]

Provide recent history (satellite imagery, NWPs, PV power, air quality, etc) unrolled (sharing weights) (i.e. one timestep at a time, each in a different cross-attention input). Then, in the last cross-attention, provide NWPs for the entire future forecast horizon (different weights).

Or maybe provide the NWPs of the future as the "latent" query; i.e. the query that's input into the whole Perceiver stack?

[JackKelly]

Try early convolutions (then attention?) https://syncedreview.com/2021/07/06/deepmind-podracer-tpu-based-rl-frameworks-deliver-exceptional-performance-at-low-cost-55/amp/

[JackKelly]

Could use the perceiver on a timestep-at-a-time. i.e. predict satellite imagery using optical flow (or Jacob's work!) and then use perceiver on each timestep (independently) to predict PV yield. The output of the Perceiver could then go into an RNN and/or self attention block that operates across the time dimension. Ie replace the CNN in the current experiment with a Perceiver.

Or feed all timesteps into the Perceiver and the predicted PV yield comes from the final output of the Perceiver.

Or could use perceiver like an RNN: each cross-attention step is a "timestep". Take the output of the self-attention block and use that for the output of that "timestep", and also feed it into the next Perceiver cross-attention. Could have multiple self attention blocks per timestep, and share weights across timestep (eg block 1 shares weights with other block 1s). Optionally, feed the output for each timestep into a self-attention block (and an RNN?) which operates across time, so forecast timesteps cal look back in time (the Perceiver only transfers information forwards in time).

[JackKelly]

0662d58a673221fd26fedf9a8b94e45305af2da6

[JackKelly]

Phew: The Perceiver is definitely learning something useful from the images (see how the predicted PV yield and the actual PV yield are well correlated in the plot in the bottom right... information about those rapid fluctuations is simply absent in the NWPs):

[JackKelly]

New Perceiver IO paper and code: https://www.google.com/amp/s/syncedreview.com/2021/08/09/deepmind-podracer-tpu-based-rl-frameworks-deliver-exceptional-performance-at-low-cost-78/amp/

Noteably, achieves state of the art in optical flow

[JackKelly]

Quoting Jacob: "The perceiver IO is now also implemented in PyTorch here: https://github.com/lucidrains/perceiver-pytorch"