[Plugin][Flytekit] Support for TFRecord as loadable schema type

[kumare3]

Why would this plugin be helpful to the Flyte community Often times users want to process data using Spark, but data is passed to a Tensorflow training process. Parquet or other columnar structures are highly in-efficient for training. To solve this problem, the TF community has done some work. It would be wonderful, if we could perform this conversion automatically depending on the context.

e.g. If the user accepts a TFRecord (data format) as a spark dataframe then we can convert, similarly if the User writes Spark dataframe, but somehow annotates it as TFRecord then we can auto-convert. Similarly, if the user reads the SparkDataframe into a process as TFRecords, we can do the conversion

This library provides this trait https://github.com/tensorflow/ecosystem/tree/master/spark/spark-tensorflow-connector LinkedIn has further updated this library to make it better in some ways https://github.com/linkedin/spark-tfrecord

Type of Plugin

• [x] Python/Java interface only plugin
• [ ] Web Service (e.g. AWS Sagemaker, GCP DataFlow, Qubole etc...)
• [ ] Kubernetes Operator (e.g. TfOperator, SparkOperator, FlinkK8sOperator, etc...)
• [ ] Customized Plugin using native kubernetes constructs
• [ ] Other

Can you help us with the implementation?

• [ ] Yes
• [ ] No

[kumare3]

This can be a good issue for beginners who understand spark, TFrecords and Flyte schemas

[github-actions[bot]]

Hello 👋, This issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏

[github-actions[bot]]

Hello 👋, This issue has been inactive for over 9 months and hasn't received any updates since it was marked as stale. We'll be closing this issue for now, but if you believe this issue is still relevant, please feel free to reopen it. Thank you for your contribution and understanding! 🙏

[github-actions[bot]]

Hello 👋, this issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will engage on it to decide if it is still applicable. Thank you for your contribution and understanding! 🙏