sghelichkhani
commented
1 year ago I've been testing the new data set by using interpolation to generate new temperature fields with standard time step for each file. However, I can't even generate new temperature fields for one file given 5 hours. There's probably some issues with the way I interpolate the data, I'll see if I can fix this.
How are you doing the interpolation? For this problem what you don't want to do is passing everything into a scipy interpolate, as it tries to load everything into memory. What I usually do in these cases I try to set up a class that builds a tree and knows the two nearest neighbors for every time that it is queried. So this would a typical skeleton for that (Note that using cKDTree is probably an overkill for this problem. But nevertheless)
import h5py
import numpy as np
from scipy.spatial import cKDTree
class HDF5Interpolator:
def __init__(self, filename):
# Load the HDF5 file
with h5py.File(filename, 'r') as file:
self.timestamps = file['timestamps'][:] # Assuming the timestamps are stored as a dataset
self.arrays = [file[f'array_{i}'][:] for i in range(len(self.timestamps))] # Replace with actual path to your arrays
# Build the KDTree
self.kdtree = cKDTree(self.timestamps.reshape(-1, 1)) # Reshape to meet KDTree input requirement
def interpolate(self, time):
# Find the two nearest timestamps
distances, indices = self.kdtree.query(np.array([[time]]), k=2)
t1, t2 = self.timestamps[indices[0]]
# Ensure the time is within the bounds of the timestamps
if t1 > time or t2 < time:
raise ValueError(f"Time {time} is out of bounds of the data timestamps.")
# Get the corresponding arrays
array1, array2 = self.arrays[indices[0][0]], self.arrays[indices[0][1]]
# Interpolate between the two arrays
alpha = (time - t1) / (t2 - t1)
interpolated_array = array1 * (1 - alpha) + array2 * alpha
return interpolated_array
# Usage:
interpolator = HDF5Interpolator('your_file.hdf5')
interpolated_array_at_t = interpolator.interpolate(5.5) # Assume we want to interpolate at time = 5.5
amartinhuertas
GiteonCaulfied
RichardScottOZ
Hi,
I've been testing the new data set by using interpolation to generate new temperature fields with standard time step for each file. However, I can't even generate new temperature fields for one file given 5 hours. There's probably some issues with the way I interpolate the data, I'll see if I can fix this.
I've also applied the new data set on the ConvAE without interpolation and result is good as expected. To reduce the computation time for testing on my local laptop, I've now moved testing code to a seperate script run by Gadi. In this case, I can simply download the testing data from Gadi to visualize it and don't have to load the 40GB data set on my local laptop before testing and visualisation. (For now this change only applies to ConvAE, will implement it for LSTM as well when I've finished training)
In the meantime, I'll test the data set on LSTM without using interpolation to see if the accuracy of the current predictions increases with this larger data set.