ARCLab-MIT / mocat-ml

A machine learning approach to space environment evolution
Apache License 2.0
1 stars 1 forks source link

Avoid using reshape in the sliding window #17

Open vrodriguezf opened 11 months ago

vrodriguezf commented 11 months ago

Right now this code:

data_sw = np.lib.stride_tricks.sliding_window_view(data, 
                                               config.lookback + config.horizon, 
                                               axis=1)[:,::config.stride,:]
samples_per_simulation = data_sw.shape[1]
data_sw = data_sw.transpose(0,1,4,2,3)
data_sw = data_sw.reshape(-1, *data_sw.shape[2:])
data_sw.shape

is inefficient because the reshape operation breaks the memory contiguity of the view. A way to avoid using it is to adapt the class DensityData, and more specifically, the method __getitem__ and __len__ so that, when the getitem is called with index idx, the corresponding subindices for the axis 0 and 1 are computed

vrodriguezf commented 10 months ago

the ideas in get_preds_iterative can be useful. The way the dataset is treated there is different than in normal training