eto-ai / rikai

Parquet-based ML data format optimized for working with unstructured data
https://rikai.readthedocs.io/en/latest/
Apache License 2.0
136 stars 19 forks source link

Rikai looks really interesting, how about integrating with flyte.org #582

Open kumare3 opened 2 years ago

kumare3 commented 2 years ago

Flyte supports custom types, that users can use natively in their code and Flyte handles the transport and maintenance of the actual data. This alleviates the complexity of using data types. Flyte is extremely type-safe and automatically provides things like memoization etc for user data.

Example of writing custom types can be found here

Some already implemented types can be seen here Check Pander, Modin, Spark, GreatExpectations etc