search

ludwig-ai / ludwig

Low-code framework for building custom LLMs, neural networks, and other AI models
http://ludwig.ai
Apache License 2.0
11.07k stars 1.18k forks source link

[Feature request] huggingface datasets #972

Open flozi00 opened 3 years ago

flozi00 commented 3 years ago

The huggingface datasets library contains very much datasets for NLP tasks, what about an native implementation of these ? Lazy loading for example is already done by the library.

w4nderlust commented 3 years ago

Unfortunately it's not a sufficiently generic solution for Ludwig as it doesn't account for datatypes other than text. That said, the library is very well done and we'll likely take inspiration from some design decisions.