Temporary failure in name resolution

[sentry-io[bot]]

This issue is affecting a lot of things:

• Redis connections in bots.law and CL
• DB connections in bots.law and CL
• Connections to Elastic (maybe)

It seems to be a network error doing a DNS lookup. Lame.

According to this post:

https://stackoverflow.com/questions/58179080/occasional-temporary-failure-in-name-resolution-while-connecting-to-aws-aurora

AWS only lets you do so many DNS requests:

Each Amazon EC2 instance limits the number of packets that can be sent to the Amazon-provided DNS server to a maximum of 1024 packets per second per network interface. This quota cannot be increased.

I'm not sure if this will fix it, but I think we need to make fewer DNS requests across our whole system.

One way to do that is to make our DB connections more persistent by tweaking CONN_MAX_AGE. Currently, it's set to 60s in CL.

Sentry Issue: COURTLISTENER-5S7

gaierror: [Errno -3] Temporary failure in name resolution
  File "redis/connection.py", line 264, in connect
    sock = self.retry.call_with_retry(
  File "redis/retry.py", line 46, in call_with_retry
    return do()
  File "redis/connection.py", line 265, in <lambda>
    lambda: self._connect(), lambda error: self.disconnect(error)
  File "redis/connection.py", line 595, in _connect
    for res in socket.getaddrinfo(
  File "socket.py", line 963, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):

ConnectionError: Error -3 connecting to courtlistener-redis.qz1axd.ng.0001.usw2.cache.amazonaws.com:6379. Temporary failure in name resolution.
(7 additional frame(s) were not displayed)
...
  File "django/core/cache/backends/redis.py", line 99, in get
    value = client.get(key)
  File "redis/commands/core.py", line 1829, in get
    return self.execute_command("GET", name)
  File "redis/client.py", line 533, in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
  File "redis/connection.py", line 1086, in get_connection
    connection.connect()
  File "redis/connection.py", line 270, in connect
    raise ConnectionError(self._error_message(e))

[mlissner]

I just set the DB_CONN_AGE to 180 in CL, and redployed it. Hopefully this will help.

[mlissner]

What's a little strange is that I don't see a spike in our traffic or anything right now, so I'm not sure why this was/is happening, but it does seem to come and go.

[mlissner]

Seems like this worked. Closing for now at least!

[sentry-io[bot]]

Sentry issue: COURTLISTENER-6PN

[mlissner]

The issue above was Redis failing to do a DNS lookup. Lame, but it got my thinking about whether our Redis client can do connection pooling. Indeed it can, so I opened https://github.com/freelawproject/courtlistener/issues/3762 to address that, and hopefully re-fix this issue.

[sentry-io[bot]]

Hm, another issue with this today:

Sentry issue: COURTLISTENER-6PV

Filed by @mlissner

[mlissner]

I think this might be fixed now that Redis and RDS are using pools. Closing for now.

[sentry-io[bot]]

another issue with this today:

Sentry Issue: COURTLISTENER-6VS

Filed by @ERosendo

[mlissner]

What bugs me about this is I haven't noticed it correlated to any kind of traffic spike, or something else that would cause more DNS requests to be made.

Bummer that our fix to RDS and Redis didn't fix it.

[sentry-io[bot]]

Related issue:

Sentry Issue: COURTLISTENER-6XS

Filed by: @albertisfu

[sentry-io[bot]]

Been awhile since I've seen this one:

Sentry Issue: COURTLISTENER-73C

Filed by @mlissner

[sentry-io[bot]]

Related: Sentry Issue: COURTLISTENER-74J

Filed by: @albertisfu

[sentry-io[bot]]

Also related: Sentry Issue: COURTLISTENER-74H

Filed by: @albertisfu

[sentry-io[bot]]

Related: Sentry Issue: COURTLISTENER-768

Filed by @albertisfu

[sentry-io[bot]]

Related:

Sentry Issue: COURTLISTENER-6Y2

Filed by: @albertisfu