[FEATURE]: Centralize envs with AWS Secrets Manager and AWS Parameter Store

[dwu359]

Feature Name

Centralize envs with AWS Secrets Manager and AWS Parameter Store

Your Name

Daniel Wu

Description

Step 1: Write a cli cmd that pulls values from parameter store/secrets manager and writes it in .env files in /training and /frontend, perhaps adding/updating an .env.example also.

Step 2: Making sure that when we deploy to prod, we delete that .env file before copying it into docker container. Whenever a container is assigned to ec2 instance, we run the cli cmd to pull the prod configs and add the env file back.

[github-actions[bot]]

Hello @dwu359! Thank you for submitting the Feature Request Form. We appreciate your contribution. :wave:

We will look into it and provide a response as soon as possible.

To work on this feature request, you can follow these branch setup instructions:

1. Checkout the main branch:

    git checkout nextjs

2. Pull the latest changes from the remote main branch:

    git pull origin nextjs

3. Create a new branch specific to this feature request using the issue number:

    git checkout -b feature-912

   Feel free to make the necessary changes in this branch and submit a pull request when you're ready.

   Best regards, Deep Learning Playground (DLP) Team

[karkir0003]

@dwu359 approach for this ticket updated?

[karkir0003]

cli: https://github.com/DSGT-DLP/dlp-cli

[codingwithsurya]

Stuff we talked about yesterday for next steps. Let me know if I missed the mark anywhere @karkir0003 @dwu359 .

1. identify what data (env variables) can go in parameter store
   • important constants:
     • prod endpoint names
       • port #
2. create that stuff in parameter store via AWS
3. build cli command for retrieving secrets from AWS secrets manager

4. DLP-CLI COMMAND SCRIPTS FOR THE FOLLOWING

   1. build .env file for frontend/
      • get the secrets toss into env file
      • projects constants added to every env file
      • include name of buckets
        2. build .env file for training/
      • get the secrets toss into env file
      • include name of buckets
   2. build .env for serverless/
      • anything in sst we wanna have as env variable
      • like dev endpoints that could go into serverless
      • include name of buckets
   3. add a new param to parameter store
   4. update existing param to parameter store
   5. remove param from parameter store

   Will probably really get most of the progress done for this Wednesday night and onwards (I have a linear algebra test on Wed). Hoping for a really productive and almost-able-to-merge PR in the next week.

[dwu359]

Yes LGTM

[karkir0003]

Fantastic detailed guide @codingwithsurya !

[codingwithsurya]

hey everyone, i have a draft PR up for this issue. Let's move all our communication for this issue there so we can have everything in one place.

[karkir0003]

@codingwithsurya I left some PR comments in your PR under dlp-cli