apple / pfl-research

Simulation framework for accelerating research in Private Federated Learning
http://apple.github.io/pfl-research/
Apache License 2.0
281 stars 27 forks source link

ArtificialFederatedDataset.from_slices can only handle 1 level of nested tuples #85

Open grananqvist opened 1 month ago

grananqvist commented 1 month ago

e.g. you might require your raw data to be of format [(Input1, input2, …), (output1, output2, …)] (2 levels of lists/tuples/dict), but ArtificialFederatedDataset.from_slices can only handle 1 level of nested tuples.

This does not work:

import numpy as np
from pfl.data.federated_dataset import ArtificialFederatedDataset
x = np.zeros(6,1)
fd = ArtificialFederatedDataset.from_slices(
data=[(x.copy(), x.copy()), x.copy()],
data_sampler=lambda length: np.random.choice(len(x), length),
sample_dataset_len=lambda: 2)
d = next(fd)

expected return dataset d with d.raw_data shape

[((2,1),(2,1)), (2,1)]