search

metatron-app / metatron-discovery

Powerful & Easy way for big data discovery
https://metatron.app
Apache License 2.0
440 stars 110 forks source link

Partial deletion of re-ingestion datasource #2186

Open jeongeunkwon opened 5 years ago

jeongeunkwon commented 5 years ago

Is your feature request related to a problem? Please describe. This is a requirement related to this issue. https://github.com/metatron-app/metatron-discovery/issues/2178

After I create and analyze a datasource, I can delete it and create a new datasource if I notice that the data in incorrect.

But if I use re-ingestion, I will not be able to delete the existing datasource. This is because I do not have all the files I need to create the datasource.

Describe the solution you'd like It would be useful to be able to delete certain parts of the datasource based on re-ingestion users or re-ingestion time If the timestamp is the current time, it may be possible to delete the segment based on the re-ingestion time, otherwise it may be more complicated. So I'm concerned it can be implemented.

Requirement received from IDCube is: a function to delete the intermediate steps to create the final version of the datasource.

I hope it will be reviewed positively.

kyungtaak commented 5 years ago

@jeongeunkwon As you know, re-ingestion is only possible with "append" or "replace" option. I think it would be more complicated to put a partial delete function here. The engine itself is not free to delete like an RDBMS. I think this feature should be better in terms of system administration. We are currently preparing engine monitoring functions. I'll review the addition inside this.