openclimatefix / power_perceiver

Machine learning experiments using the Perceiver IO model to forecast the electricity system (starting with solar)
MIT License
7 stars 1 forks source link

Topographic Data isn't appended correctly with Batch #194

Closed jacobbieker closed 2 years ago

jacobbieker commented 2 years ago

Describe the bug When generating the batches like in experiment 27, but for the production work with the preprepared batches, all the np_batch_processors work other than the Topographic one, where in xr.combine_by_coords an error of

ValueError: Every dimension needs a coordinate for inferring concatenation order

To Reproduce Steps to reproduce the behavior:

import glob
import os
from pathlib import Path

import numpy as np

from power_perceiver.load_prepared_batches.prepared_dataset import PreparedDataset
from power_perceiver.load_prepared_batches.data_sources import PV, GSP, HRVSatellite, NWP
from power_perceiver.production.model import FullModel
import pandas as pd
from torch.utils.data import DataLoader

dataset = PreparedDataset([PV(history_duration=pd.Timedelta("90 min"),),
                           GSP(history_duration=pd.Timedelta("2 hours"),),
                           HRVSatellite(history_duration=pd.Timedelta("30 min"),),
                           NWP(history_duration=pd.Timedelta("1 hour"),)],
                          data_path=Path("/home/jacob/Development/power_perceiver/data_for_testing/"))
print(dataset)
dataloader = DataLoader(dataset, num_workers=0, batch_size=None)
batch = next(iter(dataloader))

Expected behavior A clear and concise description of what you expected to happen.

Additional context Add any other context about the problem here.

jacobbieker commented 2 years ago

Using OCFDatapipes now, so closing this