Question on `aistore.pytorch.Dataset`

[insop]

Hi,

Thank you for the excellent work!

A question

• where can I see the implementation of the dataloader that is used in this?

Thank you,

    # Data loading code
    train_loader = torch.utils.data.DataLoader(
        aistore.pytorch.Dataset(
            "http://aistore-sample-proxy:51080", Bck("imagenet"),  # AIS IP address or hostname
            prefix="train/", transform_id="imagenet-train",
            transform_filter=lambda object_name: object_name.endswith('.jpg'),
        ),
        batch_size=args.batch_size, shuffle=True,
        num_workers=args.workers, pin_memory=True,
    )

https://aiatscale.org/examples/etl-imagenet-dataset/train_aistore.py

[VirrageS]

You can find the implementation here: https://github.com/NVIDIA/ais-etl/tree/master/package.

Note that the current implementation is very limited and not fully tested. We hope to extend it in the future.

[insop]

Thank you very much @VirrageS !