DNS lookups fail in Trustworthy Email scans

[jsf9k]

🐛 Summary

Some DNS lookups are failing in Trustworthy Email scans, presumably because we are hitting the 1024 queries per second per network interface throttling limit applied to AWS VPCs or another similar limit.

I noticed this in this week's BOD 18-01 scanning run, since the Trustworthy Email report for National Labor Relations Board failed to generate. When I investigated, I saw that this is because the Trustworthy Email reporting process thought there were no active domains for that agency; however, a quick dig MX nlrb.gov shows this not to be the case. Digging further, I can see many lines of this form in the file /var/cyhy/orchestrator/output/archive/latest/results/trustymail.csv:

nlrb.gov,nlrb.gov,False,False,,,,,,,,,,False,,,False,,,,False,,,,,,,,False,False,False,,[MX] In mx_scan at /var/task/trustymail/trustymail.py:95: All nameservers failed to answer the query nlrb.gov. IN MX: Server 169.254.78.1 TCP port 53 answered REFUSED,,2021-07-31T05:44:51.178821Z,2021-07-31T05:44:55.975408Z,4.796587,e044ef93-a14d-401f-981c-a9df21f3b2bb,/aws/lambda/task_trustymail,2021/07/31/[$LATEST]072fd2c0e8244d76ae42df66fcc48b74,2021-07-31T05:44:51.191Z,2021-07-31T05:44:55.971385Z,128,4.780385

Note the error message All nameservers failed to answer the query nlrb.gov. IN MX: Server 169.254.78.1 TCP port 53 answered REFUSED, indicating that the AWS DNS queries are being throttled. I thought DNS was at 169.254.169.253, but it appears that something different is happening with DNS for Lambdas live in a VPC: see here, here, and especially here. Note that the last link indicates pretty degraded performance for non-cacheable DNS requests in AWS Lambda.

This is exactly the problem that is discussed in this comment.

To reproduce

Steps to reproduce the behavior:

1. Run grep -F REFUSED /var/cyhy/orchestrator/output/archive/latest/results/trustymail.csv on the BOD 18-01 reporting instance.

Expected behavior

The Trustworthy Email results should not include any DNS refusals.

[jsf9k]

I will get an email out to our AWS contacts tomorrow morning.

[jsf9k]

I will get an email out to our AWS contacts tomorrow morning.

I sent an email to Adrian on Friday.

[jsf9k]

Adrian got back to us on August 20:

I looked through the code in the cisgov/trustymail and 18F/domain-scan repos, along with your description. It’s a sticky wicket. Here are what I see as the challenges and path forward:

Challenges:

• 18F/domain-scan doesn’t support putting the domains or the combination of the domains and the requested scan type into a queue (or queue per scan type). Nor does it have the ability to read results back from a queue or persistent store like DynamoDB. It’s limited to synchronous invocations of multiple Lambda functions.
• You’re running Lambda in a VPC. Lambda is going to use a single ENI in each subnet for the DNS queries – so all of your invocations are going to be limited to 1024 Packets per second per ENI. You only get one ENI per subnet:security group combination. (https://aws.amazon.com/blogs/compute/announcing-improved-vpc-networking-for-aws-lambda-functions/)
• I took a look through the code for dnspython (it’s a dependency). You’re using TCP for query. That’s 3 packets to establish the connection, another packet for the query, and one or more packets for the response. I looked through the code and I believe it’s going to generate a new TCP connection for each query. (I’m not an expert in this area, I could be wrong on whether it’s able to re-use the tcp connection).
• You’re running the Lambdas from within a VPC. As a result, all of the DNS queries from the Lambdas are going to go through those limited number of ENIs.

The tl;dr is what you already know:

• You’re trying to push a relatively large number of DNS packets through a small number of ENIs and getting throttled.

Options:

• Improve throughput by adding subnets to the VPC: Since the limiter is the number ENIs, adding subnets to the VPC would increase the number of ENIs. You could still get throttled, but it would be less likely.
• Reduce concurrency further: This would extend the run time, but would result in less or no throttling.
• Move to udp_with_fallback for some or all DNS queries: If some or all of your queries are less than 512 bytes, moving to udp_with_fallback would eliminate the cost of creating the tcp connection. If the queries are only one packet in size, that’s 2 packets for udp vs. 5 packets for tcp. Doing that would more than double your throughput. However, that’s dependent on whether you can determine ahead of time whether the query is likely to return less than 512 bytes.
• Move to Fargate - Refactor the domain_scan code to put the domains into an SQS queue and read the results from a separate SQS queue: Doing this gives you greater flexibility in how to perform the scans and enables you to move to Fargate. With Fargate, you get an ENI per container, as opposed to one per subnet, which provides opportunities to increase the number of DNS queries.