search

opensafely-core / backend-server

Infrastructure code for managing partner hosted OpenSAFELY backend servers.
0 stars 2 forks source link

Investigate slow and odd DNS setup for airlock #224

Closed bloodearnest closed 8 months ago

bloodearnest commented 8 months ago

Airlock is sometimes taking 5-15s to resolve DNS for jobs.opensafely.org

The container's resolve.conf has 8.8.8.8 in it, for some reason

bloodearnest commented 8 months ago

The resolve conf has 192.168.201.37 and 8.8.8.8 in it.

dig +trace jobs.opensafely.org from inside the airlock container is sad

Image

bloodearnest commented 8 months ago

Well I never.

udp/53 to 8.8.8.8 is allowed out!

Image

bloodearnest commented 8 months ago

And the 192.168.201.37 does not respond

Image

bloodearnest commented 8 months ago

I think the answer here is probably to put jobs.opensafely.org in /etc/hosts, like we did for github-proxy.opensafely.org

We should probably do that for all our DNS that resolve to dokku4

bloodearnest commented 8 months ago

And we should probably figure out where the bogus DNS resolver entry came from, and remove it

bloodearnest commented 8 months ago

Have confirmed that the windows host, ubuntu VM, and docker containers have the same DNS config, 8.8.8.8 and 192.168.201.37. The latter seems defunct and no longer works.

bloodearnest commented 8 months ago

Sent email to TPP to ask for clarification.

Suspect they changed from the 192... address to 8.8.8.8 at some point, and haven't cleaned up the old entry

bloodearnest commented 8 months ago

Ok, investigation completed.

Going to close this in favour of hardcoding DNS in /etc/hosts, as per https://github.com/opensafely-core/backend-server/issues/206

Will also remove both resolvers from Ubuntu VM's DNS config, and then ask TPP to remove 8.8.8.8 globally