search

flyteorg / flyte

Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
https://flyte.org
Apache License 2.0
5.42k stars 581 forks source link

[Core feature] [flytekit][plugin] ffcv/libffcv #3615

Open kumare3 opened 1 year ago

kumare3 commented 1 year ago

Motivation: Why do you think this is important?

It should be possible to use an ffcv datawriter and data loader and output and inputs. libffcv improves data pipelining especially for vision model training. We could use this for inference as well and accelerate both training and inferencing.

Goal: What should the final outcome look like, ideally?

TODO write an example of how the api should look

Describe alternatives you've considered

NA

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

Have you read the Code of Conduct?

github-actions[bot] commented 7 months ago

Hello 👋, this issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will engage on it to decide if it is still applicable. Thank you for your contribution and understanding! 🙏

kumare3 commented 7 months ago

Cc @fg91 @Future-Outlier have you folks looked into this

Future-Outlier commented 7 months ago

I am interested in it, can take a look at it 1 weeks later!