Issue with multiprocessing in Windows

[NoCreativeIdeaForGoodUserName]

Hi,

First of all, thank you for providing this useful set of models. Second: I think I found an issue that seems to arise with parallel processing in Windows. In main_fillmissing.py, lines 79-82:

train_loader = DataLoader(train_dataset, batch_size=_cfg.training.batch_size, shuffle=True,
                          num_workers=_cfg.training.n_cpus)
val_loader = DataLoader(val_dataset, batch_size=_cfg.training.batch_size, shuffle=True,
                       num_workers=_cfg.training.n_cpus)

Whenever "num_workers" in the Dataloaders is set to any other value than 0, this will result in the code crashing further down in the same script line 150:

for data_dict in train_loader:

Throwing an error:

... cannot pickle '_io.BufferedReader' object

This problem can be fixed by going to dataset_utils.py in line 63 to then change the line

self.data_dict = np.load(file, allow_pickle=True)

to

with np.load(file, allow_pickle=True) as data:
    self.data_dict = {key: data[key] for key in data.files}

This then properly closes the file after loading and creates a true new dictionary instance, resolving the above error.

[FranceRose]

Hi, thank you for reporting this issue. The code has not been actively tested on windows. I'll look at it this week!

[FranceRose]

I made the modification you proposed. And I tested main_fillmissing on my windows with values 0, 1, and 2 for the n_cpus parameter. It seems to work! Thank you for your help.