search

flyteorg / flyte

Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
https://flyte.org
Apache License 2.0
5.18k stars 550 forks source link

[Core feature] Rust Data Persistence #4081

Open pingsutw opened 9 months ago

pingsutw commented 9 months ago

Motivation: Why do you think this is important?

Write a Rust plugin to read/write s3 files. It can replace current data persistence plugin in flytekit.

Goal: What should the final outcome look like, ideally?

The Rust plugin should offer better data read/write performance than the current one

Describe alternatives you've considered

Use existing data persistence plugin.

Propose: Link/Inline OR Additional context

https://docs.rs/object_store/latest/object_store/

Are you sure this issue hasn't been raised already?

Have you read the Code of Conduct?

samhita-alla commented 9 months ago

@pingsutw, aren't we using fsspec instead of data persistence already?

pingsutw commented 9 months ago

Yes, we use fsspec, but we want to have a Rust data persistence plugin to replace it. Reading data using fsspec is still very slow. @troychiu is working on it.

github-actions[bot] commented 1 week ago

Hello 👋, this issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will engage on it to decide if it is still applicable. Thank you for your contribution and understanding! 🙏