Handle Too Many Open Files (Open Connections)

JohannesEbke / aws_list_all

List all your AWS resources, all regions, all services.

MIT License

414 stars 97 forks source link

Handle Too Many Open Files (Open Connections) #6

Closed JohannesEbke closed 5 years ago

JohannesEbke commented 5 years ago

If run with a nlimit of 1024, aws-list-all runs into OSErrors due to too many open files (connections).

JohannesEbke commented 5 years ago

The reason for this is that AWS has over 1000 endpoints (region/service pairs), and aws-list-all distributes requests randomly across these endpoints to avoid throttling.

Workaround: If you list regions one-by-one, you should not run out of open files.