nilsleh
commented
6 months ago @davidwilby Not sure if this helps, I ran into something similar a while ago where the target sets have different number of targets across tasks while batching, and I adapted the concat_tasks function to randomly subsample the targets to a common batch size:
for target_set_i in range(n_target_sets):
# Raise error if target sets have different numbers of targets across tasks
n_target_obs = [task["Y_t"][target_set_i].size for task in tasks]
if not all([n == n_target_obs[0] for n in n_target_obs]):
# for this target set adapt the number of observations across tasks to min_n_target_obs
shapes = [task["Y_t"][target_set_i].shape[-1] for task in tasks]
min_n = min(shapes)
for task in tasks:
rand_indices = np.random.choice(
np.arange(task["Y_t"][target_set_i].shape[-1]),
size=min_n,
replace=False,
)
task["Y_t"][target_set_i] = task["Y_t"][target_set_i][..., rand_indices]
task["X_t"][target_set_i] = task["X_t"][target_set_i][..., rand_indices]Not sure if this helps, but I would agree that it would be nice to include some functionality that handles this for batched training as this can happen quiet frequently.
tom-andersson
@tom-andersson et al. I wonder if you can help clear up some difficulty that @MartinSJRogers and I are having with batched training for gridded data with missing values using deepsensor.
As yet I'm unable to work out whether we're doing something incorrectly or whether there are bugs in deepsensor's implementation.
We're working with gridded data with missing values represented as NaNs as specified in the Data Requirements section of the docs.
When setting a
batch_sizeduring training,concat_tasksis called, in the below snippet, theremove_target_nans()method is called:https://github.com/alan-turing-institute/deepsensor/blob/aeccc097699c5963fa81b4601cfa11fad5daa41b/deepsensor/data/task.py#L477-L489
This results in a
ValueErrorraised later inconcat_taskssince there are different numbers of targets in each batch:and as a result we don't get to the calls to
mask_nans_numpyandmask_nans_npstowards the end ofconcat_tasks.I'm confused by this, since from the message above
"Cannot concatenate tasks that have had NaNs masked. " "Masking will be applied automatically after concatenation."and the later call tomask_nans_{numpy,nps}it seems like this should be handled by those methods.When I remove the call to
remove_target_nanshere for testing, of course the batch sizes are the same and theValueErrorabove isn't raised, the rest ofconcat_tasksruns andmask_nans_{numpy,nps}are called successfully.This, however, results in an error further down the line in which the
Maskedobject fromneuralprocessesis found not to have thedtypeattribute:Full stack trace
``` --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) Cell In[12], line 15 13 trainer = Trainer(model, lr=5e-5) 14 for epoch in tqdm(range(1)): ---> 15 batch_losses = trainer(train_tasks, tqdm_notebook=True, batch_size=None) # error here due to filesize. I have attempted using batch_size = n, 16 # but get seperate error asserting that the number of targets in each batch must be the same. 17 # Todo- work out how to calculate number of targets in each batch, and ensure the batch size allows me to honour this assertion. 18 losses.append(np.mean(batch_losses)) File _/deepsensor/train/train.py:177, in Trainer.__call__(self, tasks, batch_size, progress_bar, tqdm_notebook) 170 def __call__( 171 self, 172 tasks: List[Task], (...) 175 tqdm_notebook=False, 176 ) -> List[float]: --> 177 return train_epoch( 178 model=self.model, 179 tasks=tasks, 180 batch_size=batch_size, 181 opt=self.opt, 182 progress_bar=progress_bar, 183 tqdm_notebook=tqdm_notebook, 184 ) File _/deepsensor/train/train.py:145, in train_epoch(model, tasks, lr, batch_size, opt, progress_bar, tqdm_notebook) 143 else: 144 task = tasks[batch_i] --> 145 batch_loss = train_step(task) 146 batch_losses.append(batch_loss) 148 return batch_losses File _/deepsensor/train/train.py:116, in train_epoch.Are we doing something incorrectly here? Or are there bugs in the implementation. Happy to add more docs when we've worked out what we're doing or contribute bug fixes if required!
Lastly, for non-batched training, are
mask_nans_{numpy,nps}run somewhere else? I notice that they're called inmodify_taskbut I'm not yet sure when this is called.