search

duo-labs / cloudtracker

CloudTracker helps you find over-privileged IAM users and roles by comparing CloudTrail logs with current IAM policies.
BSD 3-Clause "New" or "Revised" License
887 stars 111 forks source link

"Query entered state FAILED with reason HIVE_CURSOR_ERROR: Please reduce your request rate." #63

Closed poflynn closed 4 years ago

poflynn commented 4 years ago

Looks like it is hitting an S3 API call limit which I realize is not a Cloudtracker bug per se but wondering if anyone else has hit this and if there's a workaround?

See also https://aws.amazon.com/premiumsupport/knowledge-center/emr-s3-503-slow-down/

Thanks,

Paul

INFO     Checking if all partitions for the past 12 months exist
Traceback (most recent call last):
  File "/Users/paul.oflynn/venv/bin/cloudtracker", line 11, in <module>
    load_entry_point('cloudtracker==2.1.2', 'console_scripts', 'cloudtracker')()
  File "/Users/paul.oflynn/venv/lib/python3.7/site-packages/cloudtracker/cli.py", line 104, in main
    run(args, config, args.start, args.end)
  File "/Users/paul.oflynn/venv/lib/python3.7/site-packages/cloudtracker/__init__.py", line 443, in run
    performed_actors = datasource.get_performed_users()
  File "/Users/paul.oflynn/venv/lib/python3.7/site-packages/cloudtracker/datasources/athena.py", line 317, in get_performed_users
    response = self.query_athena(query)
  File "/Users/paul.oflynn/venv/lib/python3.7/site-packages/cloudtracker/datasources/athena.py", line 74, in query_athena
    self.wait_for_query_to_complete(response['QueryExecutionId'])
  File "/Users/paul.oflynn/venv/lib/python3.7/site-packages/cloudtracker/datasources/athena.py", line 113, in wait_for_query_to_complete
    reason=response['QueryExecution']['Status']['StateChangeReason']))
Exception: Query entered state FAILED with reason HIVE_CURSOR_ERROR: Please reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown; Request ID: AA42SNIP3D63702; S3 Extended Request ID: zISuyGbB2MtV84gcJwjpvoSNIPI5WyUM+C8Ln+XcxSNIPgmL3Jsmi/EJYYFQdW9s=)
(venv) cloud-tracker $
0xdabbad00 commented 4 years ago

Interesting. I haven't seen something similar. I'm curious if you find out what the limit it is that it is hitting, like if someone has 100GB of CloudTrail logs, should they expect to hit this issue?

poflynn commented 4 years ago

Interesting. I haven't seen something similar. I'm curious if you find out what the limit it is that it is hitting, like if someone has 100GB of CloudTrail logs, should they expect to hit this issue?

Dunno to be honest but I was able to work around this by specifying a 'start date' so that's good enough for me for now. Thx for the reply.. Closing..