search

amphi-ai / amphi-etl

Python-based Low-code ETL for data manipulation and transformation. Generates Python code you can deploy anywhere.
https://amphi.ai
Other
793 stars 35 forks source link

Pipeline metadata #56

Open simonaubertbd opened 3 months ago

simonaubertbd commented 3 months ago

Hello,

it would be nice, for data governance, to have some metadata on our pipelines : -author -copyright -title -key words -a long description -ability to add a link for documentation -version

Best regards,

Simon

simonaubertbd commented 2 months ago

We can even imagine a tool "amphinception" who parses the pipelines to retrieve the metadata. I love when a tool is able to analyze its own assets.

tgourdel commented 2 months ago

Definitely, I'm currently weighing in the best way to achieve this. For now, everything configurable about a pipeline is through components that you can drag and drop. I'm thinking it could be the same for metadata (would be easier for me and maybe for the users as well). The only thing I don't like with this approach is it also clutters the pipeline view. In any case, the metadata would be present in the pipeline DSL in JSON, which can be read by Amphi: image

simonaubertbd commented 1 month ago

Oh, and of course, it should be exported to generated python code (with an option to not export)