DNS resolution fails on 2022.09 image

mud5150 commented 2 years ago

Problem

I am trying to build a new container image to use on CircleCI, so my Dockerfile starts with

FROM cimg/base:2022.09

I am using the Docker orb default publish job to attempt to build my new image. During the build process any command that requires network access fails with getaddrinfo() thread failed to start, which appears to be a DNS failure within the running context of the container build. To verify this I did some testing by SSHing into the hosted runner and manually executing commands against container instances .

Testing

When I start a container instance from the current 2022.09 convenience images on a CircleCI hosted runner, processes within the container are unable to perform dns lookups. Other images, including cimg/base:2022.08 don't have this issue. You can easily verify this by running docker run --rm -it cimg/base:2022.09 /usr/bin/curl google.com from within the hosted runner.

Interesting data points

This only happens on the CircleCI hosted infra. If I run this same container image on my local system there is no problem with dns lookups. I am running Ubuntu 22.04 locally.
The docker engine or run method has no effect. I tested with both a machine runner and several different docker versions by specifying the use-remote-docker option in the orb.
I didn't see anything different in /etc/hosts on the 2022.09 image

Examples

These examples were collected by SSHing into a machine based runner

Fails

circleci@ip-172-28-40-102:~$ docker run --rm -it cimg/base:2022.09 /usr/bin/curl google.com
curl: (6) getaddrinfo() thread failed to start

Works

circleci@ip-172-28-40-102:~$ docker run --rm -it cimg/base:2022.08 /usr/bin/curl google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

mud5150 commented 2 years ago

I also found this problem to be present in last month's Azure image cimg/azure:2022.08

FelicianoTech commented 2 years ago

Looking into this.

FelicianoTech commented 2 years ago

The ability to use DNS seems to be working fine on the image. I ran some tests here: https://app.circleci.com/pipelines/github/felicianotech/cci-testing-pub/802/workflows/8523e1fa-0413-478b-9c53-4601b9289458/jobs/1814

Perhaps this is an issue with the Docker orb rather than this image. The Docker orb is using a fairly old image for the executor. Also, starting with the September base image, by default Ubuntu 22.04 is used instead of 20.04. The default executor for the orb is 20.04.

In the short term, try using 2022.09-20.04 and see if that helps? In the long-term, I think this is likely a Docker orb issue. I will move this issue over to that project.

brivu commented 1 year ago

Hey @mud5150,

Thank you for opening this issue and providing us with detailed examples. I went ahead tested the publish job with a dockerfile starting with FROM cimg/base:2022.09. I subsequently ran multiple curl and wget commands in the Dockerfile to test the network access. I was able to run the job successfully and had no issues.

I am going to close out this issue for now but if this still persists, please feel free to re-open.

Thanks again!

Best, Brian

CircleCI-Public / docker-orb