search

uber / petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
Apache License 2.0
1.78k stars 285 forks source link

String as input in petastorm dataloaders #795

Open freud14-tm opened 1 year ago

freud14-tm commented 1 year ago

In the _sanitize_pytorch_types function used in petastorm dataloaders, an exception is raised when an input is a string:

https://github.com/uber/petastorm/blob/0b0775af42539189d913702d9695566431dabd8a/petastorm/pytorch.py#L64

Is this really necessary? Maybe the default_collate function from PyTorch might not always like, although it seems to support it according to the documentation, but this prevents making other types of collate functions.

Would you consider removing the exception? Thank you.

selitvin commented 1 year ago

Sure! Would you like to propose a PR for that? Ideally, with a small unit test showing that it works.

freud14-tm commented 1 year ago

It seems more complicated than I thought with the current code base of the library and I am not using the library right now. Still think the idea is worthwhile but feel free to close the issue if you wish. Otherwise, let it open in order to keep the idea close.

pratikgujjar commented 1 year ago

Wondering if there is a reason for the string type to not be supported? A column in a parquet file could be all string labels. It is currently not possible to read this parquet file as a PyTorch dataloader becuase this line in the code.