awslabs / aws-serverless-data-lake-framework

Enterprise-grade, production-hardened, serverless data lake on AWS
https://sdlf.workshop.aws/
MIT No Attribution
417 stars 140 forks source link

Compatibility install option for SDLFv2 #281

Open cnfait opened 8 months ago

cnfait commented 8 months ago

Having SDLFv1 and SDLFv2 side-by-side in the same accounts. [more details to follow]

cnfait commented 7 months ago

290 helps handling the CodeCommit part of this issue.

cnfait commented 7 months ago

291 helps handling the DynamoDB part - but there is more work required.

cnfait commented 7 months ago

295 and #296 are also linked to this task.

cnfait commented 7 months ago

There is a branch with the changes necessary to install SDLFv2 side-by-side with SDLFv1: https://github.com/awslabs/aws-serverless-data-lake-framework/tree/sdlf1-backcompatibility

Some of the changes have been incorporated into main (see previous comments) but not all are relevant/desirable for all installations of SDLFv2. This is an upgrade path more than a good solution long-term.

Important caveats:

cnfait commented 7 months ago

Other changes that may become part of main:

removing FunctionName for the replicate lambda in sdlf-foundations https://github.com/awslabs/aws-serverless-data-lake-framework/commit/c968be52b3b3be3c4fedc02951ab5c8b719c2704

retrieving the Glue crawler name from SSM in sdlf-stageB fetch-metadata Lambda function: https://github.com/awslabs/aws-serverless-data-lake-framework/commit/612e15454100680edeff42db786d7febb5b03f1e