architecture change

[Jean-Baptiste-Lasselle]

• the transformers: The transformers, must not work with files anymore. They must only work on polars dataframes. Just one transform method, the source dataframe, the resultDataframe, and that's it, period.

• The ingesters (Extract) : will ingest the datasets from data sources, and return a polars dataframe

• The loaders (Load): they will load a dataset from a source datastore, and persist

This taks is about re-designing the components, one thing is sure, the trasnformers must only run operations on polars dataframes, and must not know about the details of how to persist the dataframe into the database, like not knowing what is the name of the table itself (although the datarframe knows about the columns, and their associated types.

[Jean-Baptiste-Lasselle]

https://docs.lakefs.io/understand/how/merge.html

[Jean-Baptiste-Lasselle]

https://docs.lakefs.io/integrations/git.html