Multi-image loading in Imagefolder dataset

[vvvm23]

Feature request

Extend the imagefolder dataloading script to support loading multiple images per dataset entry.

This only really makes sense if a metadata file is present.

Currently you can use the following format (example metadata.jsonl:

{'file_name': 'path_to_image.png', 'metadata': ...}
...

which will return a batch with key image and any other metadata.

I would propose extending file_name to also accept a list of files, which would return a batch with key images and any other metadata.

Motivation

This is useful for example in segmentation tasks in computer vision models, or in text-to-image models that also accept conditioning signals such as another image, feature map, or similar. Currently if I want to do this, I would need to write a custom dataset, rather than just use imagefolder.

Your contribution

Would be open to doing a PR, but also happy for someone else to take it as I am not familiar with the datasets library.

[mariosasko]

Supporting this could be useful (I remember a use-case for this on the Hub). Do you agree @polinaeterna?

Implementing this should be possible if we iterate over metadata files and build image/audio file paths instead of iterating over image/audio files and looking for the corresponding entries in metadata files.

[dennisrall]

I've build a similar feature from scratch and would be interested to combine it with the datasets package.

My solution works something like this: Interpret the first element of each column as a file path. If the path exists and is a file, (try to) load the files for the entire column. Thereby, one isn't restricted to a particular column name, with comes in handy when dealing with multiple file columns.

I've looked into the code to try to implement this, but didn't find the right places. I'm also open to contribute, but will need some guidance.

[severo]

Required here: https://discuss.huggingface.co/t/dataset-repo-requires-arbitrary-python-code-execution/59346/14

[whydna]

+1

Is the only way to do this right now to write a custom dataset loader script?

[severo]

Also: be able to have input and output images for each row. Asked here: https://discuss.huggingface.co/t/how-to-structure-image-files-for-datasets-load-dataset-imagefolder-when-you-have-input-and-output-images-like-in-instruct-pix2pix/82467