Upgrade Orca from v8.1.0 to v9.0.5

[krisstanton]

Upgrading Orca from v8.1.0 to v9.0.5

• Add Correct Version numbers to this ticket and update the title (We need a MINIMUM of ORCA v9.05 in order to get the HTTPS update)
• Reference Load Balancer Tickets
  • https://github.com/NASA-IMPACT/csdap-cumulus/issues/344
  • https://github.com/NASA-IMPACT/csdap-cumulus/issues/345
• Add Research lines from the upgrade notes, right here, to this ticket description

Upgrade Steps (After any Code and/or Migration Changes)

• [x] Run make all-init
  • [x] DOTENV=.env.sandbox make all-init
• [x] Run make all-up-yes
  • [x] DOTENV=.env.sandbox make all-up-yes (or if just deploying only the Cumulus module: up-cumulus-yes)
• [x] Run smoke test in sandbox
• [x] Open PR
• [x] Get a soft approval from reviewers
• [x] Make the server side changes (see steps below)
• [x] Trigger UAT Deployment
• [x] Run smoke test in UAT
• [x] If smoke test works in UAT,
• [x] Merge Code
• [x] Make server side changes in PROD
• [x] Deploy to PROD

Current Cumulus and Orca Version information

- Cumulus Version:     v18.1.0  // Verified by: https://github.com/NASA-IMPACT/csdap-cumulus/blob/main/config/helpers/cumulus_version_helper.rb
- ORCA Version:         v8.1.0  // Verified by: https://github.com/NASA-IMPACT/csdap-cumulus/blob/3153f8f00194f8c5c3f658a938b9963bf1c55440/app/stacks/cumulus/orca.tf#L18   // source = "https://github.com/nasa/cumulus-orca/releases/download/v8.1.0/cumulus-orca-terraform.zip"
- Terraform Version:     1.5.7  // Verified by: https://github.com/NASA-IMPACT/csdap-cumulus/blob/3153f8f00194f8c5c3f658a938b9963bf1c55440/.terraform-version#L1
- Postgresql Version:   11.21   // Verified by: Checking CBA PROD Account: 5047 RDS DB Clusters --> Configuration --> Engine Version:   // https://us-west-2.console.aws.amazon.com/rds/home?region=us-west-2#database:id=cumulus-prod-rds-serverless;is-cluster=true;tab=configuration

References Current Cumulus Version: v18.1.0

• https://github.com/NASA-IMPACT/csdap-cumulus/blob/main/config/helpers/cumulus_version_helper.rb Current Orca Version: v8.1.0
• https://github.com/NASA-IMPACT/csdap-cumulus/blob/3153f8f00194f8c5c3f658a938b9963bf1c55440/app/stacks/cumulus/orca.tf L18

Cumulus Upgrade Research Reference

• https://github.com/nasa/cumulus/releases

Link to the last Cumulus upgrade ticket

• https://github.com/NASA-IMPACT/csdap-cumulus/issues/281

---

(Steps to make SandBox deployment work successfully) -- Developing the Steps here - Following for Orca Upgrade from v8.1.0 to v9.0.5

• [ ] Upgrade Orca from v8.1.0 to v9.0.5 (sandbox account 7894)
  • [x] v9.0.4 Required Changes complete on sandbox
  • [x] Modify Package location
  • orca.tf // line 18 Change v8.1.0 to v9.0.5
  • [x] Manually remove All Lambda functions with the tag "Application"=="ORCA" (19 total in sandbox 7894)
  • This is to satisfy fix in v9.0.4
  • [x] Modify / Remove some specific security groups related to ORCA
  • Navigate to the AWS console and search for the Cumulus RDS security group. (I just went to EC2 and clicked Security Groups)
  • [x] Remove the inbound rule with the source of PREFIX-vpc-ingress-all-egress in Cumulus RDS security group.
    • Mine on Sandbox was called, "cumulus-kris-sbx7894-vpc-ingress-all-egress"
    • Removed this inbound rule: " – sgr-05e1e5f935f8ef0a6 PostgreSQL TCP Allow PostgreSQL ingress for cumulus-kris-sbx7894 Orca lambdas. "
  • [x] Search for PREFIX-vpc-ingress-all-egress and delete the security group NOTE: Due to the Lambdas using ENIs, when deleting the security groups it may say they are still associated with a Lambda that was deleted by the script. AWS may need a few minutes to refresh to fully disassociate the ENIs completely, if this error appears wait a few minutes and then try again.
    • Mine on Sandbox was called, "cumulus-kris-sbx7894-vpc-ingress-all-egress"
    • Removed this whole group. " - sg-05285e4b891d25dbc cumulus-kris-sbx7894-vpc-ingress-all-egress vpc-03fbe3759df14c423 ORCA security group to allow PostgreSQL access. 772632177894 0 Permission entries 2 Permission entries"
    • If there was an error, make sure the above step where the ORCA related lambdas need to be deleted is complete and that enough time has passed (a few minutes) so that AWS knows there is no longer an association.
    • [x] There was one more change I had to make (that was not clear from the instructions) (before I could remove that above group)
    • I had to go into this security group: "cumulus_rds_cluster_acess_ingress20230213165434378100000001"
    • And remove the inbound rule that had this info in it " sgr-01915d2b3ad2395dc PostgreSQL TCP Allows cumulus-kris-sbx7894 Orca lambda access. "
  • [x] v9.0.5 Required Changes completed on sandbox
  • [x] need to delete the load balancer target group before deploying this version
    • NOTE (about the below two steps): I had a lot of trouble with this section (couldn't delete the rule because it was in use by the target, couldn't delete the target because it was in use by the rule - In the end I just deleted the listener first. Redeployment should bring it back).
    • My Sandbox Listener: was the only listener on the load balancer.
    • My Sandbox Target Group: arn:aws:elasticloadbalancing:us-west-2:772632177894:targetgroup/09736c6c-gql-a/deeb2320302aca34
  • [x] From AWS EC2 console, go to your load balancer named and select the Listeners and rules tab. Delete the rule. (IMPORTANT, before deleting the rule, open your target group in a new browser tab that is linked in the rule - it is easier to get to it this way!)
    • If you only have the one rule, than before you can delete the rule decribed above, you have to make a new rule and just leave it blank
  • [x] Delete your target group -gql-a. The target group name has been randomized to avoid terraform resource error.
  • Verify This After Deployment: When Orca is deployed, if deployed correctly, the target group health checks should show as healthy.
  • [x] Successful sandbox deploy to 7894
  • [x] Successful Smoke test on Sandbox Deploy
  • [x] Had to make one additional change to fix a failed smoke test (TODO for UAT and PROD deployments)
    • on the DR Archive bucket, Change policy: Remove line "s3:x-amz-acl": "bucket-owner-full-control"
    • Also remove the comma on the line before it

Note: If after deployment, ORCA Does not work (verified by smoketest), Do not forget about the aws_ssm_parameter (orca_s3_secret_key) (Where we have to put in keys in the secure parameters list in order to keep it functioning)

[krisstanton]

WIP Update: Still researching. There was a bug during the last upgrade. After some digging and a good bit of reading, I found that the deployment bug between Cumulus 18.1.0 and Orca 9.0 may now be fixed. So an ORCA 9.0.5 upgrade should work after following some of the other specific instructions in the release notes. Will be trying a sandbox deploy soon.

[krisstanton]

WIP Update: I was able to get Orca successfully deployed on the Sandbox environment. Thanks Chuck for the assist with the lint timeout issue and export command. We found that if the yarn test:lint process takes too long, the process is killed and the deploy fails. This can happen if the CPU of the machine which is doing the deploy is being too over worked.

We also had to add an extra line of code in the docker file for exporting an environment variable for checking the version of terraspace: ENV TS_VERSION_CHECK=0 Next up is to run the smoke test and verify everything.

The Cumulus Upgrade requires a database upgrade. I split out the Cumulus upgrade from the Orca upgrades into two separate tickets. This is the current ticket is for the ORCA Upgrade while the Cumulus upgrade is now covered in ticket: #358

[krisstanton]

WIP UPDATE: Sandbox deployment success. There was an error when running the smoke test. The fix for the error was as follows:

On the DR Archive bucket, Change policy: Remove line "s3:x-amz-acl": "bucket-owner-full-control"
Also remove the comma on the line before it

Note: We will also need to make this change on the PROD DR bucket during PROD deployment (before sandbox)

Next up will be to make the Pull Request, after soft approval, make the necessary changes on UAT, then click Deploy to UAT, run smoke test on UAT, if all goes well, then make the changes on PROD and then do a code merge to deploy to PROD

[krisstanton]

WIP Update: The ORCA upgrade is now deployed in UAT with a passing Smoke Test.

Next step is to make the changes on the PROD sever and then deploy the upgrade there too.

@chuckwondo Hey Chuck, if you need to upgrade your Sandbox Deployment to keep up with the rest of them. Here is a shortcut link to the PR which contains the instructions that you have to do within AWS via manually clicking on the AWS dashboard (the instructions at this link are the ones I did for UAT and are a little more of a condensed version of what is in this ticket above). https://github.com/NASA-IMPACT/csdap-cumulus/pull/359

[krisstanton]

Orca Upgrade Deployment to PROD is now complete!