ESPv2 image vulnerabilities

[husseyd]

Guys - love ESPv2. Simple concept and so flexible.

With that said, as part of my team's CI pipelines we scan all container images for vulnerabilities. We're pretty vigilant about this with 0 critical and 0 high vulns tolerated.

ESPv2 serverless for Cloud Run shows a huge number of vulns from Snyk container scan. gcr.io/endpoints-release/endpoints-runtime-serverless:2.30.1 3 critical, 31 high, 51 medium, 274 low

This is really a product of the base image being a fully fledged Debian Buster distro. I've not gone through the process to try running this on a lighter distro like buster-slim or ideally Alpine yet, but I can imagine that a fully bloated Debian distro might not be the minimum requirement to run ESPv2 / Envoy. Understand that glib/musl is a challenge.

Switching to a slimmer image could bring security benefits but also performance - current release image is 1.32GB!

tl;dr - thoughts on a slimmer more secure base image like Alpine or Ubuntu-core?

[qiwzhang]

We will like to have a slimmer base image. But it requires some work to test them out. Currently, we don't have resources for this. We welcome any contributions.

[nareddyt]

+1 to not having resources. I remember we did some study on this internally a few months ago.

I poke around alpine image. It is too small, it doesn't have libc libraries we need

ldd /bin/envoy
        /lib64/ld-linux-x86-64.so.2 (0x7f2c6229b000)
        libm.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f2c6229b000)
        librt.so.1 => /lib64/ld-linux-x86-64.so.2 (0x7f2c6229b000)
        libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f2c6229b000)
        libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f2c6229b000)
        libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f2c6229b000)
Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /bin/envoy)
Error relocating /bin/envoy: __cmsg_nxthdr: symbol not found
Error relocating /bin/envoy: fcntl64: symbol not found
Error relocating /bin/envoy: pthread_getname_np: symbol not found
Error relocating /bin/envoy: __cxa_thread_atexit_impl: symbol not found
Error relocating /bin/envoy: __strftime_l: symbol not found

We deprioritized the task since then.

[husseyd]

No worries guys. I'll hopefully try it out myself on another base image in the next week or two. Will let you know how it goes.

[johnpena]

@husseyd any luck with this? We're facing the same issue - base image has several vulns that we're uncomfortable putting into production.

[nareddyt]

We actually changed the base image to alpine in https://github.com/GoogleCloudPlatform/esp-v2/pull/611. It eliminates all these vulnerabilities, as mentioned in the PR description. The next release should resolve this issue.

[husseyd]

Awesome! 🥳

[husseyd]

Nice work