SFTP Production Setup

[somesylvie]

Story

As a partner, so that I can use the Intermediary, I need the infrastructure, code, and secrets in production.

Acceptance Criteria

• [x] Validate that the most recent code are deployed to production
• [x] Validate that secrets have been updated in production to the real values
• [x] Validate that infrastructure was created
• [x] Validate that the function app function was created
• [x] We can test with partners in Prod
• [x] The SFTP Ingestion Service sends messages to RS as the ca-phl.etor-nbs-results sender, not flexion.simulated-lab (the scope in code and the CA_PHL_CLIENT_NAME in terraform will both need new values).

Tasks

Research

• [ ] Research work needed to complete the story
• [ ] Foundational: High-level research that will support this and future efforts

Engineering

• [x] Set up production TF environment
• [x] Set up Github prod deploy workflow
• [x] Rename network security group (RITM0343277 service desk ticket created) - @halprin
• [x] Make sure CRON is set to not run in production and that the function app function doesn't do 'run once on startup' - it's currently set to run on Mondays in May, which should be fine
• [x] Update scope to be correct for CADPH

Edit: as of 8/21, we've decided to use the ca-phl org that's been created and is in staging already (and is ETOR/TI specific), rather than doing anything with the covid-specific ca-dph that already existed

• [x] Rename Flexion-flavored env vars and/or secrets to use ca-phl as part of the name to create a consistent pattern

  • [x] Rename
  • [x] Rename in terraform
  • [x] Rename in mock credentials
  • [x] Rename in code
  • [x] Rename in docker file
  • [x] update renamed values in dev after deploy
  • [x] test
  • [x] update renamed values in internal after deploy
  • [x] test
  • [x] update renamed values in staging after deploy
  • [x] test

• [x] Keys to access ReportStream (as ca-phl.etor-nbs-results) - as of 9/16, Jorge and James got the private key to put into ca-phl-reportstream-private-key-{env}. Prod's secret update will be tracked in a task down below.

  • [x] Update code to use ca-phl.etor-nbs-results as the sender in app.tf.
  • [x] Update the scope in report_stream_sender.go to be ca-phl.*.report.
  • [x] Update the ca-phl-reportstream-private-key-internal secret in internal
    • [x] ensure sending to RS works.
  • [x] Update the ca-phl-reportstream-private-key-dev secret in dev
    • [x] ensure sending to RS works.
  • [x] Update the ca-phl-reportstream-private-key-stg secret in stg
    • [x] ensure sending to RS works.

• [x] Rename keys.

  • [x] Rename keys associated with our user's credentials.
  • [x] ca-phl-sftp-private-key-env to ca-phl-sftp-user-credential-private-key-env.
  • [x] Also make sure there is a public key paired with the private key in the mock_credentials.
  • [x] Reference both these keys in docker-compose.yml.
  • [x] Rename keys associated with CDPH's SFTP server.
  • [x] ca-phl-sftp-public-key-env to ca-phl-sftp-host-public-key-env.
  • [x] Also make sure there is a private key paired with the public key in the mock_credentials.
  • [x] Reference both these keys in docker-compose.yml.
  • [x] Delete any extra keys in Terraform and in mock_credentials.
  • [x] Add a README.md to document the types of keys and the naming conventions, etc. (either create a new one and link from the main one, or update an existing one that makes sense)

• [x] Consider moving files around keybase to better align with naming convention. Renaming as needed perhaps. - @halprin

No longer blocked by CADPH API key setup:

• [x] Turn on using-API-keys-for-SFTP in code once CADPH has set that up on our prod and non-prod user

• [x] Deploy to production (this has been done once, but needs to be done again after the name changes above have been merged)

• [x] After Terraform is run in production, update all keys/passwords/secrets that need manual updates:

  • [x] Keys/secrets to access CA SFTP external site
  • [x] ca-phl-reportstream-private-key-prd
  • [x] ca-phl-sftp-host-public-key-prd
  • [x] ca-phl-sftp-server-address-prd
  • [x] ca-phl-sftp-starting-directory-prd
  • [x] ca-phl-sftp-user-credential-private-key-prd
  • [x] ca-phl-sftp-user-prd
  • [x] ca-phl-zip-password-prd

• [x] Update RS key for other envs. Update code to use the correct sender.

• [x] Triage prod and why we can't dequeue the SFTP trigger message.

• [x] Have Ott make org in RS prod.

• [x] Test in prod.

• [x] Set CRON to February 30th (or whatever so it doesn't run) until our partners are ready.

Definition of Done

• [x] Documentation tasks completed
  • [x] Documentation and diagrams created or updated
  • [x] Implementation guide (/ig folder)
  • [x] ADRs (/adr folder)
  • [x] Main README.md
  • [x] Other READMEs in the repo
  • [x] If applicable, update the ReportStream Setup section in README.md
  • [x] Threat model updated
  • [x] API documentation updated
• [x] Code quality tasks completed
  • [x] Code refactored for clarity and no design/technical debt
  • [x] Adhere to separation of concerns; code is not tightly coupled, especially to 3rd party dependencies
  • [x] Code is reviewed or developed by pair; 1 approval is needed but consider requiring an outside-the-pair reviewer
  • [x] Code quality checks passed
• [x] Security & Privacy tasks completed
  • [x] Security & privacy gates passed
• [x] Testing tasks completed
  • [x] Load tests passed
  • [x] Unit test coverage of our code >= 90%
• [x] Build & Deploy tasks completed
  • [x] Build process updated
  • [x] API(s) are versioned
  • [x] Feature toggles created and/or deleted. Document the feature toggle
  • [x] Source code is merged to the main branch

Research Questions

• Optional: Any initial questions for research

Decisions

• Optional: Any decisions we've made while working on this story

Notes

• Should we maybe come up with a production testing proposal. I feel like this will definitely come up the closer we get to moving CA to prod. Something like the morning we cutover to production, we will send 3 production messages with test characters 1, 2, and 3 to confirm transport and validation rules are working. Maybe 2 messages that would pass and one that should fail validation.

[jherrflexion]

Notion doc: https://www.notion.so/flexion-cdc-ti/Prod-Setup-c44be771f74242069909ae16c3a77039

[somesylvie]

Code is merged, but unable to deploy because someone changed permissions for our github deployer. We're blocked on deploying to staging and prod, and thus also blocked from updating the keyvault secrets until this is resolved

[scleary1cs]

Looks like we have the information to reset the "third" SFTP account password:

Please have them go to https://selfservice.cdph.ca.gov Click "Reset Password" on the left-hand side Enter the username TESTFlexionUCSDSS01, leave domain as External Partner, enter characters and click Continue Follow the steps to create a new password. Please have them bookmark this site. Passwords expire after 90 days.

@somesylvie can we pair on this? Want to make sure this gets captured in Keybase.

[scleary1cs]

TESTFlexionUCSDSS01 password has been reset and added to keybase.

[scleary1cs]

keybase://team/cdc_ti/secrets_to_access_partner_services/CA/cdph-sftp-nonprod-credentials.txt

[scleary1cs]

New keys sent to Sumitha:

Sumitha, We've posted updated keys in RSA format. Please let us know if they work for you/Lauren.

Path to the files for each account: /Home/CDPH-SIS-Staging/HIE-Staging/UCSD

[scleary1cs]

Follow up poke sent to Sumitha on 08/23/24 at 2:40 PM EDT.

[scleary1cs]

Update request on UCSD - Secure Keys for SFTP sent to Jamie Matterson. Jamie will follow up with Sumitha regarding updated key pairs.

[scleary1cs]

Looks like the SSH keys are all set: Hi Shawn, I got confirmation from ITSD that the keys were added successfully to the respective service accounts. We should be all set. Thanks Sumitha

[scleary1cs]

Sumitha/Jamie, We are seeing some odd behavior. We are able to log in to both production and staging accounts using the keys we shared, another key, and also no key. We were only expecting logins to work with the correct keys. Can you verify these security settings have been turned on? Best, Shawn

[scleary1cs]

From Sumitha:

Sambathkumar, Sumitha@CDPH Sumitha.Sambathkumar@cdph.ca.gov 12:32 PM (16 minutes ago) to Lauren@CDPH, me, cdc-ti, Jamie@CDPH

+Lauren,

Hi Lauren – can you please assist. See Shawn’s comment below. Is there something in addition we need to configure? Thanks Sumitha

[jherrflexion]

This is ready for merge. Hoping to test in Internal tomorrow.

[jherrflexion]

We are running into frequent issues uploading keys due to Azure Portal quirks. Specifically the whitespace and multiline pastes not being supported.

We've resorted to uploading the file into Teams, and using VIM to add the new line within Azure Portal.

[jherrflexion]

Tasks have been updated above. PR is merged. We plan to tackle a good chunk of the remaining work today.

[jherrflexion]

Production deploy and secrets are complete.

[jherrflexion]

Prod testing successful