NielsRogge
commented
4 weeks ago Hi,
In case your dataset consists of images, labels and corresponding JSONs, the easiest might be to start from here: https://huggingface.co/docs/datasets/image_dataset (based on a folder containing your images organized by labels).
Alternatively, this could be useful: https://github.com/huggingface/transformers/tree/main/examples/pytorch/semantic-segmentation#creating-a-datasetdict. Basically you start from local lists, then cast the image column to the Image feature.
A third option could be to start from a Pandas dataframe (where the JSON data is stored as a dict for each row), then use:
from datasets import Dataset
dataset = Dataset.from_pandas(df)
dataset.push_to_hub("your-hf-username-or-org/abc-123")Let me know if you need any help
mahobley
Hello @mahobley 🤗
I'm Niels and work as ML engineer at Hugging Face. I discovered your work through ECCV (we met at your poster presentation!), and indexed your paper here: https://huggingface.co/papers/2309.04820. The paper page lets people discuss about your paper and lets them find artifacts about it (your dataset for instance) you can also claim the paper as yours which will show up on your public profile at HF.
Would you like to host the datasets on https://huggingface.co/datasets? I see you're using your own servers for it. Hosting on Hugging Face will give you more visibility, and will also allow people to do:
If you're down, leaving a guide here: https://huggingface.co/docs/datasets/image_dataset. We also support Webdataset, useful for image/video datasets: https://huggingface.co/docs/datasets/en/loading#webdataset.
Besides that, there's the dataset viewer which allows people to quickly explore the first few rows of the data in the browser.
After uploaded, we can also link the datasets to the paper page (read here) so people can discover your work.
What do you think?
Kind regards,
Niels