Open thomasjpfan opened 1 month ago
The proposed simplified TypeTransformer API for data backed by FlyteFile
or FlyteDirectory
aims to make it easier for users to define their own serializers and deserializers without needing to learn the full TypeTransformer
API. This approach would streamline the process by focusing on the essential methods for saving and loading data.
According to the TypeTransformer documentation, creating a custom transformer involves implementing methods like get_literal_type
, to_literal
, and to_python_value
. The proposed API simplifies this by only requiring save
and load
methods, which handle the serialization to and from FlyteFile
.
Would you like more detail on how to implement this simplified API or examples of existing TypeTransformer
implementations?
Sources:
This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.
Mention @runllm to ask follow-up questions. See runllm.com for more!
Motivation: Why do you think this is important?
Currently, creating a type transformer requires learning about the TypeTransformer API which requires someone to learn about Flyte Literals, Flyte Context, etc. For example, here is the TypeTransformer for
np.ndarray
For the simple case, most data are backed by just a
FlyteFile
orFlyteDirectory
at rest. I propose we have a simplified interface for creating these type transformers. This way, we can make it very easy for users to define their own serializers and deserializers.Goal: What should the final outcome look like, ideally?
I propose this simplified TypeTransformer API for data backed by FlyteFile:
The same API can then be applied to
FlyteDirectory
.For example, we can define the NumPy Transformer as:
Describe alternatives you've considered
Do nothing and only have
TypeTransformer
API.Propose: Link/Inline OR Additional context
No response
Are you sure this issue hasn't been raised already?
Have you read the Code of Conduct?