Closed cliang-huanglab closed 2 months ago
Hi @cliang-huanglab 👋
Alas, it is a bit tricky to shuffle data from an IterableDataset
. However, you can invoke the samples
argument as part of the **kwargs
to shuffle data as it is a LanceDataset
. This shuffles the data using a reservoir sampling approach.
However, we're in the process of updating Lance under the hood, which should make a few things better!
When i run these codes
I get a error.
How to use shuffle in Dataloader?