search

ml6team / fondant

Production-ready data processing made easy and shareable
https://fondant.ai/en/stable/
Apache License 2.0
341 stars 25 forks source link

Create a generic component that combines load from parquet and load from hub #475

Closed PhilippeMoussalli closed 1 year ago

PhilippeMoussalli commented 1 year ago

Related to https://github.com/ml6team/fondant/pull/474#pullrequestreview-1648769479

RobbeSneyders commented 1 year ago

As discussed offline, I don't think we should merge these. The LoadFromHub component should be able to download datasets from HF hub in different formats. We can either have a separate LoadFromParquet component, or merge it with the LoadFromFiles component.

PhilippeMoussalli commented 1 year ago

ok closing this