search

lhotse-speech / lhotse

Tools for handling speech data in machine learning projects.
https://lhotse.readthedocs.io/en/latest/
Apache License 2.0
904 stars 204 forks source link

Question about WebDataset #1285

Closed Ryu1845 closed 4 months ago

Ryu1845 commented 4 months ago

Hello, I'm interested in using WebDataset with Lhotse, but I was wondering why all the data is pickled rather than using JSON + pickle, which seems to be more standard. I can implement it myself and submit a PR if it's a problem of manpower.

pzelasko commented 4 months ago

I recommend to instead take a look at our own webdataset inspired format that can store multiple types of features and metadata in separate tar archives/JSON files: https://github.com/lhotse-speech/lhotse/blob/master/examples/04-lhotse-shar.ipynb

I appreciate your offer to contribute but I’m afraid that changing the existing format would break the functionality for existing users.

Ryu1845 commented 4 months ago

I understand, that's unfortunate. I was interested in WebDataset because of its standardization and its ability to be streamed from a remote location, so I don't think I'll use shar. Thank you for the suggestion. I think I'll write something on top of the library instead of integrated into it.