Replace 3rd-party Docker images with a solution to pass SecRel

[yoomlam]

Our current redis and rabbitmq images cause SecRel to fail. The SecRel team are allowing them to pass temporarily for our deployment. Replace these 3rd-party Docker images with Bitnami

Slack thread:

Option 2: You can explore the option of switching to an alternative image from Bitnami which runs on a rootless container by default and will pass the policy without any changes needed. For option 2, those are available here: https://hub.docker.com/r/bitnami/rabbitmq/ https://hub.docker.com/r/bitnami/redis/

(If Bitnami images can be retrieved from elsewhere, then we could avoid the DockerHub rate limit problem.)

We should also use Bitnami images as the base image for our other containers. This will be handled in a separate ticket.

Why? Replace docker images with the hope of reducing SecRel alerts over time. Will need a few months to measure the success of this approach. Bitnami was recommended by our App Assessor.

Acceptance Criteria

• 3rd-party Docker images (redis + rabbitmq) have been replaced with Bitnami wherever possible/feasible (if unsure, ask the team!)
• The system is able to be deployed better or at least as well as before this change

[yoomlam]

• https://bitnami.com/stack/redis/containers
• https://bitnami.com/stack/rabbitmq/containers

[sethdarragile6]

Consider using an ubuntu 21.04 based rabbitmq image to address CVE-2023-0286

[sethdarragile6]

Also keep in mind CVE-2023-0361, which says that libgnutls30 on mirror-redis should be 3.7.1-5+deb11u3 or greater. Acknowledged in Aqua @greene858

[yoomlam]

Other considerations before moving forward:

• https://itnext.io/which-container-images-to-use-distroless-or-alpine-96e3dab43a22
• https://www.reddit.com/r/kubernetes/comments/lojf76/comment/go98nfz/?utm_source=share&utm_medium=mweb3x
• https://airman604.medium.com/cve-analysis-of-popular-base-images-b43dc939c582

Given that many developers have Apple Silicon chips, the chosen the base image should have a arm64 tag to facilitate local testing.

[dianagriffin]

From chatting with @greene858: we think this ticket could be broken down into more bite-sized pieces that could provide some good starting points for Raghu/Mason once they've got sufficient access to start working on tickets. e.g. A proof of concept ticket, etc. (Tom, hoping you can fill in the blanks where I missed the details of what the potential ticket breakdown might look like)

[dianagriffin]

Relevant slack thread on distroless images compared to Bitnami's images

[yoomlam]

A good first target is to update https://github.com/department-of-veterans-affairs/abd-vro/blob/develop/svc-bgs-api/src/docker/Dockerfile#L1 which is only used by this one Gradle project.

[yoomlam]

Trying bitnami: update Python base image

[yoomlam]

Flyway (db-init) doesn't offer a Bitnami image: https://github.com/flyway/flyway-docker#supported-tags

[greene858]

Bitnami currently is running Debian 11 (bullseye) which has known vulnerabilities. There is no timeline for this upgrade. I moved our python image back to Alpine. We can revisit this idea later. I also looked into distroless, which is promising; but belongs in a separate ticket. I will discuss with our assessor and create a ticket if needed

[dianagriffin]

This issue has resurfaced as a blocker to deployments now that SecRel requires signatures on third-party images. Slack discussion in thread here, key comment copy/pasted for discussion/prioritization of this ticket:

The debian issues might be fixed now but even so those vulnerabilities can be acknowledged and unblock deployment where as our current images have a super user which cannot be bypassed at all. a switch to bitnami is the easiest way to do that, there are other options but require publishing our own custom versions of the images we currently use and maintaining those

[nelsestu]

One option to consider would be to open a conversation with LHDI about whether it is possible to replace our self hosted containers with their AWS managed counterparts. Amazon MQ supports RabbitMQ and Amazon ElastiCache supports Redis. By running these services as self-hosted containers we are taking on a significant burden of responsibility and will always be subject to this type of deployment blocking issue. Meanwhile, migrating to AWS hosted services would be a permanent fix, that reduces our overall operations responsibilities. Another possible plus side is that if we invest our time expanding LHDI capabilities vs identifying the next workaround, doing so successfully could enable teams throughout the VA to also benefit from that work, thus having a significant impact across the VA as a whole. If this approach has been discussed and rejected in the past, I am happy to bite my tongue on the subject, but I thought it is worth suggesting.