BUG: Twistlock Authentication Failure

[mjones-oddball]

Hello Twistlock tenants,

The DOTS team has upgraded twistlock to the latest version: 22.06 and have migrated all of your data.

As we stated before in previous coms, some data would not migrate and you were advised to export the data for your records. We have exported everyone’s old data just in case, and can assist you if you need it for archiving purposes. Alternately, the old environment is still active at this URL here: twistlock-dte.dots-ftl.com (this is a temp DNS that does not have a valid cert, so just ignore the cert warnings in chrome) so you can still do the export yourselves/ensure your new environment has all the expected data. We will be deleting the old environment on: 12/11/22

Finally, the new defender NLB address has also been updated and you will need to update your deployed defenders to use the new address.

New NLB address: DOTS-PROD-Defender-ELB-8be52ee5f6b5930e.elb.us-gov-west-1.amazonaws.com

After the above upgrade, we are locked out of our account. We cannot log in through LDAP, nor can our github actions execute the Twistlock scans on deployments to AWS, so we're locked out of testing in Dev.

• [x] Work with DOTS to restore Twistlock access
• [x] Document root cause in case this recurs
• [x] Revert changes of PR 987
• [x] Delete the "no-twistlock" and "with-twistlock" work-around branches

[k-macmillan]

Reached out to Jared Piimauna. My Twistlock password needed reset. That allowed me to login but the github actions are still failing. Reached out to Jared again with more details including direct links to our failing actions and the code that runs them.

We seem to have all our credentials correct (all logins work), but we don't see authorized to run the scan.

[k-macmillan]

Jared got back to me with a password update for our github action. Updated in parameter store. Didn't fix it. Stopped and started the instance, didn't fix it. Reached out to Jared again and he responded with another suggestion to change --project VaNotify to --project vanotify. I will have to create a branch and then use that branch to test the action change.

[k-macmillan]

Note: Added new checklist item to revert the PR that was just linked.

[kalbfled]

Here's what I think I know at this point . . .

There are 3 VMs/containers/instances involved:

1. The Github Actions job runner for twistlock.yml. This is Ubuntu.
2. The Twistlock instance visible in the AWS EC2 console. This is not something we build. I think it is given to us by another team.
3. The container in ECR that we want to deploy.

When we trigger a deployment action, the Notify API is packaged as a container and deployed to ECR. Subsequently, the job runner issues a twistcli command that triggers the Twistlock EC2 instance to scan the new build container in ECR. This works because the Twistlock instance has the credentials necessary for this access.

I'm told that the problem is that we need to upgrade the version of twistcli. It is not obvious to me how this CLI is getting installed on the job runner in the first place. I would not expect it to be part of the Ubuntu VM Github provides. If somebody can answer this question, we should be able to move forward.

[jessecanderson]

There is the Twistlock EC2 instance is where the twistcli should be installed and not anything in GitHub. The GitHub runner just pushes up the commands to the instance using the uses: ./.github/actions/run-commands-on-ec2 and then waits for the output from that instance. If you need to upgrade the twistcli then it should be done on that EC2 instance in AWS. I would assume that you can console to it the same way you could console to the other machines. But I don't know for sure.

[cris-oddball]

Based on Jesse's comment, is the EC2 configured to use the correct IAM ? We just had this problem with Locust and had to change the Locust file to use the correct IAM. Right now, the twistlock instance fails 1/2 checks and the console throws a permissions error when trying to connect to it.

[kalbfled]

There is the Twistlock EC2 instance is where the twistcli should be installed and not anything in GitHub. The GitHub runner just pushes up the commands to the instance using the uses: ./.github/actions/run-commands-on-ec2 and then waits for the output from that instance. If you need to upgrade the twistcli then it should be done on that EC2 instance in AWS. I would assume that you can console to it the same way you could console to the other machines. But I don't know for sure.

@jessecanderson Thank you for that information. The "-on-ec2" part of that went over my head until then.

[cris-oddball]

@kalbfled @k-macmillan a few other data points:

• There is something on the twistlock instance that fills up the logs (see Cloudwatch for the twistlock utility) and overwhelms the insatnce. This is most likely why the twistlock instance needs to be fully stopped and then fully restarted
• On the twistlock ec2 instance, in usr/bin (the default directory when logging in) there is a twistcli binary - this is what needs to be updated. The current version of this file on the EC2 instance is 20.09.345 - we should be on 22.06

The best way to update twistcli is as follows: @ldraney

• Create an S3 bucket (maybe call it 'twistlock')
• Set perms on the bucket and/or the EC2 instance such that the EC2 instance can receive and send files to/from the bucket
• upload the file to the S3 bucket using the AWS console interface.
• log into the EC2 instance via the AWS console and copy the file from the S3 bucket to usr/bin, overwriting the existing file (which you might want to give an .old extension to) aws s3 cp s3://{bucket_name}/{filename}

Once updated, there are still changes needed:

The output from the failed action indicates an authorization error.

----------ERROR-------
Status: 401 Unauthorized
GET https://twistlock.devops.va.gov/api/v1/version?project=VaNotify failed.

• Verify with Jared/Kyle that the AWS ssm parameter store contains the correct password for the vanotify-ci-user
• Note that the VA is now running version 22.06 and that the API has been changed such that if you call /api/v1/ then you get the most recent code (which is not 22.06) so the API call must replace v1 with the version in use.
• Jared also notes that VaNotify should be changed to vanotify in the above URL
• Note that their API docs do not mention any query parameters using ?project on the /version route but in theory, based on Jared's email, the route to call should thus be https://twistlock.devops.va.gov/api/v1/version?project=vanotify
• It appears the call for this is in /.github/workflows/twistlock.yml lines 44-52 but I do not see how it is constructing the url to use v1so need to learn how to construct the url to use v22.06
• Probably best to just get Jared live on a call, zoom or teams, to work through how to actually make the correct call.

[cris-oddball]

@kalbfled @k-macmillan @ldraney alternate method to get the twistlock cli on the server without using an S3 bucket:

See docs here for the download client method: https://prisma.pan.dev/api/cloud/cwpp/22-06/util#operation/get-util-arm64-twistcli

In case the above call requires authentication https://prisma.pan.dev/api/cloud/cwpp/22-06/authenticate#operation/post-authenticate

[k-macmillan]

• Regarding the log issue: Increased volume and cron job to reboot?
• The updated password has been put into parameter store
• I could find no reference to "v1" when trying to fix what Jared mentioned regarding the url. Still not sure where that gets set
• VaNotify was changed to vantofiy and it failed sooner (I think). It no longer displays the successful login message
  • That one needs to be done with the 22.06 update I believe, they don't work separately

A call with Jared or someone else on his team would definitely be beneficial at this point.

[cris-oddball]

from dsva va-notify-team channel from Filip Fafara last December, this upgrade didn't seem to happen until late 2022.

Got notification from DOTS team that they are going to upgrade twistlock during off-hours on Dec 11th. We will need to upgrade our twist-cli. I think the best way to approach this is to just rebuild our twistlock image. It should download correct version of the CLI from twistlock console as part of cloud-init.

Edit to remove question about the ECR image since that image is of our app and not relevant.

[cris-oddball]

Found it! That URL is set in this twistlock-init.cfg

and twistlock-init.cfg is invoked from twistlock.tf

runcmd:
  - systemctl enable amazon-cloudwatch-agent
  - systemctl start amazon-cloudwatch-agent
  - rpm --rebuilddb
  - yum -t -y install docker
  - usermod -a -G docker ssm-user
  - systemctl enable docker
  - systemctl start docker
  - [ sh, -c, "sed -i 's/put_password_here/'$(aws --region us-gov-west-1 ssm get-parameter --name /utility/twistlock/vanotify-ci-user-password --with-decryption | jq '.Parameter.Value' -r)'/g' /root/.netrc" ]
  - [ curl, -k, -L, -n, -o, /usr/bin/twistcli, "https://twistlock.devops.va.gov/api/v1/util/twistcli?project=VaNotify"]
  - chmod +x /usr/bin/twistcli

[cris-oddball]

This issue is tested and resolved with these two PRs that are awaiting approval, merge and deploy.

Infra PR

• this is the root cause. when they upgraded twistlock, we needed to update the client on the EC2 server. This is done by changing the URL to the twistcli. They also changed our password, so that was updated in the AWS SSM Parameter Store API PR
• this PR reverts PR 987 such that the twistlock scan runs again and changes the project name to lowercase vanotify.

Also ignored all found vulnerabilties until jan 15. Will need tickets to handle those.