reata / sqllineage

SQL Lineage Analysis Tool powered by Python
MIT License
1.3k stars 235 forks source link

delete some intermediate tables #633

Open gyafdxis-k opened 3 months ago

gyafdxis-k commented 3 months ago

Whether we try to delete some intermediate tables. Deletion using regular matching, such as tables starting with ^tmp can be deleted in the blood relationship. The upstream and downstream of tables starting with ^tmp can be directly connected according to the Cartesian product.

1234

gyafdxis-k commented 3 months ago

del some intermediate tables function but I think this feature can be config

delphisharp commented 3 months ago

tmp or temp is a business definition. I don't think it's a good idea to introduce business definitions. Business processing should be done at the business level. What if my temporary table definition is based on the schema name? Or other logic?

delphisharp commented 3 months ago

Is there any problem in processing business logic on the parsing results? I mean, a tool should have its own boundaries, so I don't think this business processing should be done at the tool level.

If you want to do it, you should use an interface or other more compatible methods. Scenarios I encountered

  1. Create Session Table in teradata
  2. Schema is Tmp or ods_temp, is temp table In this case, I process the parsing results at the business level.

For the current metadata_provider, I also think that providing a callback function is enough, and how to implement it is up to the developer to handle it.

gyafdxis-k commented 2 months ago

mp or temp is a business definition. I don't think it's a good idea to introduce business definitions. Business processing should be done at the business level. What if my temporary table definition is based on the schema name? Or other logic?

It is only a example, I think the callback function is a good idea which is friendly to the user thanks for your replying, I think the sqlineage is very good opensource project