Create generic shares_base and s3_datasets_shares modules from current dataset_sharing

[dlpzx]

Is your feature request related to a problem? Please describe. It is difficult to add new sharing item types with the current implementation of the datasets_sharing module.

Describe the solution you'd like If we want to add more types of sharing(e.g. Redshift dataset sharing, notebook sharing) a first step will be the creation of a generic class that defines a data.all Shares abstraction that can be used by each of the particular sharing implementations.

Describe alternatives you've considered This are rough design considerations that might change during implementation:

---

⚠️ WIP This item is a continuation of https://github.com/data-dot-all/dataall/issues/1123 that implements the top part of the diagram, the separation between datasets_base and s3_datasets. Now we need to do the same for shares.

Backend

1. shares_base module

With the generic code to create share objects and share items.

• api - all APIs on ShareObject and the approval/reject/revoke workflow. List generic ShareItems
• aws - nothing, it should be technology agnostic
• db - generic ShareObject and ShareItem models. Operations on these models
  • in ShareObject: rename datasetUri as targetUri
  • in ShareItem: remove Glue and S3 details from the shareItem
  • repositories: all operations that do not involve details on S3/Glue
• cdk - nothing, it does not involve any particular AWS IAM permissions or infra
• handlers - ECS handler that handles shares tasks and executes sharing manager methods accordingly
• services
  • sharing_service - interfaces to implement ShareProcessors
  • shares_enums - they are generic - ShareItemTypes will be used to know the items that can be shared
  • shares_permissions - they are generic
  • share_object_service
  • share_item_service
• tasks
  • generic shareTask that loads all shareProcessors
  • generic shareVerifier that loads all shareProcessors
  • generic shareReapply that loads all shareProcessors

2. s3_datasets_shares module

With the specific code to share S3 dataset items.

• api - TBD: if there is any API that is specific to s3 datasets it will remain here
• aws: the same ones as before, lakeformation, s3, glue, kms...
• db
  • Specific repositories to get share items with details of Tables, Folders, Buckets
• cdk
  • pivot role and env role permissions
• handlers: nothing
• services: the specific services used to share S3 resources in the ECS task
  • processors
  • managers
  • other utilities
• tasks: no sharing tasks! the actual sharing ECS task is generic, the specific logic is imported through services

Frontend

Most views related to shares are very generic. If anything, share item tables and share details (consumption details) might be s3-specific. Since the shares view is being modified in other workstreams this is paused at the moment.

Config.json

tbd

[dlpzx]

Step-by-step implementation plan

• [X] Rename dataset_sharing to s3_datasets_shares and create shares_base module (PART1)
• [X] Move enums and permissions to shares_base(PART1)
• [X] Move ShareObject and ShareItem db models to shares_base (PART2)
• [X] Move Share state machines to shares_base into its own file (PART2)
• [X] Create Interface in shares_base to execute sharing tasks from s3_datasets_shares (PART3)
• [X] Same for Share-verify tasks (PART3)
• [X] Same for Share-reapply tasks (PART3)
• [X] ~Create base ShareItem model in shares_base. Use ShareItem to create S3ShareItems or GlueShareItems, S3BucketShareItem, S3AccessPointShareItem~ --> Instead of making multiple ShareItems with particular information about the shareItem (glutable...) we should be using the itemType and itemUri to get the table technical data. The task is now: remove the usage of shareItem.GlueTableName, GlueDatabaseName and AccessPointName (PART4)
• [X] Move generic support files to shares_base: exceptions and notifications (PART5)
• [X] Split APIs into generic APIs and s3-specific APIs (PART6)
• [X] Make share object service independent from s3 datasets and move to shares_base (PART7)
• [X] Make share item service independent from s3 datasets and move to shares_base, create a second share_item service in s3 if needed (PART8)
• [X] Split db-repositories (PART9)
• [X] Remove any other S3 reference in shares_base (PART10)
• [X] Rename files and clean-up s3_datasets_share (PART11)

The next items are enhancements

• [ ] Ensure bullet-proof error handling: https://github.com/data-dot-all/dataall/issues/1266
• [ ] Move share notifications parameters in config.json to shares module
• [ ] Comment in https://github.com/data-dot-all/dataall/pull/1298 --> make dataset lock acquire/release a context
• [x] Comment in https://github.com/data-dot-all/dataall/pull/1312 --> review typing in functions: use S3Dataset, RedshiftDataset or DatasetBase accordingly

[dlpzx]

The core functionality has been completed. There will be follow-up enhancements related to shares tackled in separate github issues