[New Process Improvement Need]: LocalStack starter kit

[galenatjpl]

Checked for duplicates

Yes - I've already checked

Category

Software Lifecycle - the creation, change, and release of software

Describe the need

Lately it has been demonstrated (e.g. Drew's presentation, and usage on SRL) that LocalStack has a lot of potential in future development efforts. I'm proposing that we create a starter kit and/or best practices surrounding LocalStack. For example, if I'm starting a new project/mission, and I know I'll be developing on the cloud, it would be nice for a CI/CD-integrated local dev environment including LocalStack, with perhaps a few example services and terraform setup scripts to be included. That way I could hit the ground running, and start practicing local development from day one. We would probably want to reach out to Drew and/or Stirling to get more specific setup/configuration ideas for this, but overall I think there would definitely be an audience for this type of setup, if SLIM could streamline this.

[riverma]

Thanks for the ticket @galenatjpl! I really like the idea for this. Maybe we could brainstorm some use cases:

1. Running an existing Terraform based cloud IaaS deployment locally
2. Kicking off a fresh LocalStack setup for a basic EC2/S3/Lamda example
3. (@jl-0) Using LocalStack to find errors / problems in deployment early

Also - it'd be great to think how we could leverage SLIM's Standards-as-Code principle with this idea. That's something I want to discuss more with @NASA-AMMOS/slim-tsc in general.

[stirlingalgermissen]

Here’s the recording of Drew's presentation: https://jpltube.jpl.nasa.gov/watch?v=POtCNS

Slides are attached to that link recording

[ddalton-swe]

@riverma @galenatjpl If we have an interested project, we could attempt to convert their existing Terraform configuration to Localstack as a test case. I think we could also figure out a sample workflow that many projects use and have that as an example

[riverma]

@ddalton-jpl - per the conversation with @PaulMRamirez:

1. It'd be good to explore DataDrive as a use case (CC @rtapella)
2. I think it'd be super useful to see an architecture diagram (or flowchart) describing how the proposed starter kit would work - essentially how projects can benefit quickly through the following use cases:
   • My project has an existing Terraform-based AWS deployment
   • My project does not have any AWS deployment and is starting from scratch

In summary: can we make a simple tool or recommendation for existing projects to easily leverage LocalStack if they're already deeply vested in testing on AWS? Also for new projects? How do we make that (1) simple, (2) infusible per our SLIM infusion strategy.

[yunks128]

@ddalton-jpl - thank you for working on LocalStack, which can potentially save a lot of cloud costs.

I would like to have a feature in the starter kit that enables me to filter which AWS projects can benefit the most from LocalStack. For instance, only the LocalStack Pro version offers support for SageMaker. Besides, since LocalStack utilizes local resources, I assume that tasks requiring high computation (such as deep learning) may not work efficiently with LocalStack. What do you think?

[ddalton-swe]

@yunks128: Thanks for your feedback!

Some of the information related to supported services can be found here. That assumption is correct. For LocalStack, you are limited to the resources on your device.

Although he isn't a SLIM developer, I want to keep @drewm-jpl in the loop on this. He is a major reason why people became interested in LocalStack at JPL.

As we determine the plan for implementation and infusion, I wanted first to lay out the overall goal. We want to investigate how we can move projects that utilize cloud infrastructure to test and run their resources locally before moving onto a cloud environment. We want to do this for a few reasons:

1. Deploying test resources to a development environment is expensive and requires coordination from multiple people to configure if it is used by a team.
2. The iterative testing and improvement process can be slowed significantly if there are issues with permissions.
3. The risk of security issues during development is reduced.

[ddalton-swe]

I also wanted to link this presentation. It covers all of the documentation and motive for why local development is important. The hope for this ticket is to avoid the collection of documentation and instead find a solution that allows projects to adopt local cloud development easily.

[riverma]

Some suggestions from conversations with @jl-0 @PaulMRamirez:

Use cases

• DataDrive's existing AWS deployment (using Terraform deployments) to support local deployment testing
  • Generalization: easily converting an existing terraform-deployment to local-stack
• DataDrive extensions (e.g. AOCS) should be able to be included in cloud deployments
  • Generalization: easily creating terraform-compatible local-stack configurations that add on to an existing deployment
• Best practices for tagging, costing, etc. included in local-stack configuration

FAQ requests

• How to have multiple apps running locally in local stack that use same port

Audience

• Users: @jl-0 @sjoshi-jpl

Suggested next steps

1. Design a time-saving approach to quickly allow a developer (e.g. a DataDrive developer) to accomplish above two use cases
2. Share approach on this ticket and get feedback from @jl-0 @sjoshi-jpl @PaulMRamirez among wider community
3. Begin implementing approach for DataDrive use case, but written in a general way

@ddalton-jpl - see recommendations on how to write a SLIM guide here for a fresh reminder: https://nasa-ammos.github.io/slim/docs/contribute/submit-best-practice