search

data-dot-all / dataall

A modern data marketplace that makes collaboration among diverse users (like business, analysts and engineers) easier, increasing efficiency and agility in data projects on AWS.
https://data-dot-all.github.io/dataall/
Apache License 2.0
228 stars 82 forks source link

Add resource lifecycle rules #732

Open degoldner opened 1 year ago

degoldner commented 1 year ago

Is your feature request related to a problem? Please describe.

In order for the data.all product to be adaptable by different users with different requirements we need to make the features be modularizable. This was greatly already achieved on the level of datasets, data pipelines and further features.

Another important layer which would require modularization possibilty is the governance layer where we define our permission model and the business workflow for governance, like data curation and data share request approval. The requirements for the governance layer might differ from user to user.

Additional context For our example we adapted the data.all code (not yet on new modularized version) by introducing a RBAC model with specified roles and permission. Also we added a curation process which is in between dataset import and dataset publishing to the data catalog.

zsaltys commented 11 months ago

@anmolsgandhi more details what is being planned here would be good

dlpzx commented 11 months ago

This issue has been added to v2.0.0 for further investigation. At the moment there are no details on the actual implementation because we are still gathering requirements. However, the main goal is to allow a more configurable definition of the lifecycle of a dataset. @degoldner and his team have implemented an extra stage for datasets plus more customization on permissions. Once they are imported/created they need to be published before being shareable in the catalog.

In this issue we want to dive deep into their customizations and investigate/implement a way in which we could add that feature to data.all v2 in a generic way that could serve all customers. Once we start the discussions I will make sure to keep you in the loop @zsaltys.

dlpzx commented 10 months ago

@degoldner we are studying this feature as a candidate for our next releases. Can you provide more details, screenshots, diagrams on your design?

dlpzx commented 1 month ago

This issue will be closed soon as stale issue