4dn-dcic / tibanna

Tibanna helps you run your genomic pipelines on Amazon cloud (AWS). It is used by the 4DN DCIC (4D Nucleome Data Coordination and Integration Center) to process data. Tibanna supports CWL/WDL (w/ docker), Snakemake (w/ conda) and custom Docker/shell command.
MIT License
70 stars 28 forks source link

botocore client error Rate Exceeded #381

Open nhartwic opened 1 year ago

nhartwic commented 1 year ago

Basically, any tibanna stat command that would return more than about 75 results produces an error.

Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/envs/snake/bin/tibanna", line 8, in <module>
    sys.exit(main())
  File "/home/ubuntu/miniconda3/envs/snake/lib/python3.9/site-packages/tibanna/__main__.py", line 576, in main
    subcommandf(*sc_args)
  File "/home/ubuntu/miniconda3/envs/snake/lib/python3.9/site-packages/tibanna/__main__.py", line 502, in stat
    API().stat(sfn=sfn, status=status, verbose=long, n=nlines, job_ids=job_ids)
  File "/home/ubuntu/miniconda3/envs/snake/lib/python3.9/site-packages/tibanna/core.py", line 483, in stat
    parse_exec_desc_and_ec2_desc(exc['executionArn'], verbose)
  File "/home/ubuntu/miniconda3/envs/snake/lib/python3.9/site-packages/tibanna/core.py", line 407, in parse_exec_desc_and_ec2_desc
    exec_desc = client.describe_execution(executionArn=exec_arn)
  File "/home/ubuntu/miniconda3/envs/snake/lib/python3.9/site-packages/botocore/client.py", line 508, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ubuntu/miniconda3/envs/snake/lib/python3.9/site-packages/botocore/client.py", line 915, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the DescribeExecution operation (reached max retries: 4): Rate exceeded

I'm guessing AWS made some change to table access rates or something?

alexander-veit commented 1 year ago

Which Tibanna version are you using? I assume this was working previously? Did you switch Tibanna versions?

I can see that you are using Python 3.9 which is not supported officially (although it most likely works). Did you install Tibanna from source?