Static embedding

[gabrieltseng]

On the way to a static embedding.

Right now, it's just a one hot encoding (which is sort of like letting the ea-lstm learn a 128 dim embedding per location, since 128 = hidden size), but it would be nice to have a linear layer before that to have a smaller embedding.

• [x] Add a spatially based one hot encoding
• [x] Get it working for the EA-LSTM with an embedding

[tommylees112]

One of the experiments we can run is to cluster the output (the 128 dimension embedding) to see if the model is learning the spatial structure of the points (like they do in the Kratzert et al paper)

[gabrieltseng]

So there is a "true" static embedding now, if the EALSTM is run with

include_yearly_aggs=False, 
static="embedding", 
include_pred_month=False

(but it works if those are not the options too - a linear layer is just stuck in front of the static input before it's passed to the recurrent part of the network).

[tommylees112]

Looks Peng mate well done and thank you :D :D

[tommylees112]

Can we also write the output of the one hot encoding (i know it's really simple it's literally an ID for each pixel) but i just want to make sure that the data being read in to the models is exactly as expected. How do you do this when you want to check? Via the tests?

This is something i want to know for debugging the nowcast experiment too

[tommylees112]

Also @gabrieltseng can we add something into the load_model function that will allow older models to be loaded? Or do you think it's not necessary?

In [21]: ealstm = load_model(model_path)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-21-4fec1710b1d5> in <module>
----> 1 ealstm = load_model(model_path)

/data/ml_drought/src/models/__init__.py in load_model(model_path, data_path, model_type, device)
     78             init_kwargs[key] = val
     79
---> 80     model = str_to_model[model_type](**init_kwargs)
     81     model.load(**model_dict["model"])
     82

TypeError: __init__() got an unexpected keyword argument 'include_static'

https://github.com/esowc/ml_drought/blob/01b4ff3cb8de2146cfe8a197238df01ce94b18b6/src/models/__init__.py#L80

That init_kwargs dictionary could just have the value for 'include_static' popped? I don't know what you think about this in general (might not be necessary ..)

[gabrieltseng]

Can we also write the output of the one hot encoding (i know it's really simple it's literally an ID for each pixel) but i just want to make sure that the data being read in to the models is exactly as expected. How do you do this when you want to check? Via the tests?

This is something i want to know for debugging the nowcast experiment too

This can be achieved just by taking the output of the dataloader, and saving it (or using it however) - we do this in the plot_explanations file in the analysis folder

Also @gabrieltseng can we add something into the load_model function that will allow older models to be loaded? Or do you think it's not necessary?

Hmm yea this definitely breaks backward compatibility. I would prefer not to, since many things can still change and maintaining it will be a headache in the future, but if this will make things way harder for you, I am happy to give it a shot