search

huggingface / candle

Minimalist ML framework for Rust
Apache License 2.0
15.84k stars 957 forks source link

Any plan to enhance candle dataset? #1439

Open npuichigo opened 11 months ago

npuichigo commented 11 months ago

It's still too simple compared to huggingface/dataset. I wonder if there's any plan or roadmap to enhance that?

LaurentMazare commented 11 months ago

Actually I'm working on some new dataset api that will be a bit independent of candle so likely to live outside of this project. It's already in a workeable state but will require at least a couple weeks before we're able to open-source it so I don't think there would be much plans in the candle-dataset front in the meantime.

chenwanqq commented 9 months ago

Maybe you can have a look at my candle-distributed. The whole project is at very early stage but I have already implemented a pytorch style dataloader. Currently I have done multi workers and pre-fetch (pin-memory not yet).

npuichigo commented 9 months ago

@chenwanqq it seems the link is unavailable

chenwanqq commented 9 months ago

@chenwanqq it seems the link is unavailable

fixed hahaha