App Seeding Module (Seed Code, CodeCommit, CICD Pipeline)

[JunjieTang-D1]

Do we need a system design doc for App Seeding module?

[AndyEfaa]

@a13zen Do you mean the creation of a sample pipeline that can deploy ADDF from a defined manifest file with AWS native tooling such as AWS CodeCommit, AWS CodePipeline and AWS CodeBuild? If yes, the concept is ready. It's a matter of prioritization.

[a13zen]

The idea of this is to have a simplified "workbench" like module that allows deploying custom applications that have their own lifecycle separate from the core ADDF one, but still able to interface. For example, a team developing custom monte-carlo simulation containers.

We can create an ADDF module that would deploy a very simple service catalog product template that contains a CodeCommit Repository, CodePipeline + CodeBuild hooked into the repository (CDKPipelines candidate). Use cases can spin up a new service catalog product that provisions these resources and they can then develop their application in their own lifecycle. Applications that are common and should form a baseline e.g. if the team developed a pipeline to also build and push images to ECR, it can be brought back into the template as seed code so the next use case/team can start from that point onwards.

This gives users 2 powerful things:

1. A way to deploy Service Catalog Products via ADDF
2. A way to develop reusable app templates for different "plug in" use cases

[AndyEfaa]

The idea of this is to have a simplified "workbench" like module that allows deploying custom applications that have their own lifecycle separate from the core ADDF one, but still able to interface. For example, a team developing custom monte-carlo simulation containers.

We can create an ADDF module that would deploy a very simple service catalog product template that contains a CodeCommit Repository, CodePipeline + CodeBuild hooked into the repository (CDKPipelines candidate). Use cases can spin up a new service catalog product that provisions these resources and they can then develop their application in their own lifecycle. Applications that are common and should form a baseline e.g. if the team developed a pipeline to also build and push images to ECR, it can be brought back into the template as seed code so the next use case/team can start from that point onwards.

This gives users 2 powerful things:

1. A way to deploy Service Catalog Products via ADDF

2. A way to develop reusable app templates for different "plug in" use cases

That means we load module code from a third party source. This could be Service Catalog , this could be anything else. Theoretically, we can already do that right now by building a deployspec that does exactly that with native AWS API calls. Or do you want to abstract that away in the manifest files? Example for service catalog:

name: servicecatalog-dynamodb
type: servicecatalog
productid: "prod-12lscsnkyha34"
parameters:
  - name: table-name-admin
    value: admin

[a13zen]

The template seed code can be from a third party source. The module only creates the service catalog product and wires together the initial configuration for the application developers to start working from. The idea is we can create templates that encapsulate multiple complex components together that can be deployed in a self service manner.

This is the same concept/model used by SageMaker Project Templates, and indeed the same ADDF module can create custom SageMaker Project Templates too (since they are just service catalog products with special tags).

[srinivasreddych]

@a13zen and @AndyEfaa Is this still relevant?

[dgraeber]

@AndyEfaa @a13zen Has this changed?

[a13zen]

The sagemaker project mlops module implements this solution