fyoorer / ShadowClone

Unleash the power of cloud
Apache License 2.0
672 stars 86 forks source link

Rate limit error while running command #14

Closed b1bek closed 1 year ago

b1bek commented 1 year ago

Any solution for this ?

(env) kkkk:ShadowClone:% python shadowclone.py -i subdomains.txt --split 300 -o httpxresults.txt -c "/go/bin/httpx -l {INPUT} -t 100"                                                                   <main ✗>
2022-10-20 04:52:29,805 [INFO] Splitting input file into chunks of 300 lines
2022-10-20 04:52:29,869 [INFO] Uploading chunks to storage
2022-10-20 04:52:29,886 [INFO] lithops.config -- Lithops v2.5.8
2022-10-20 04:52:29,928 [INFO] lithops.storage.backends.aws_s3.aws_s3 -- S3 client created - Region: us-east-1
2022-10-20 04:52:30,062 [INFO] lithops.serverless.backends.aws_lambda.aws_lambda -- AWS Lambda client created - Region: us-east-1
2022-10-20 04:52:30,064 [INFO] lithops.invokers -- ExecutorID 85b642-0 | JobID M000 - Selected Runtime: lithops_v2-5-8_7vdl/sc-runtime1 - 512MB
2022-10-20 04:52:37,246 [INFO] lithops.invokers -- ExecutorID 85b642-0 | JobID M000 - Starting function invocation: execute_command() - Total: 167 activations
exception calling callback for <Future at 0x7fedac13f9d0 state=finished raised Exception>
Traceback (most recent call last):
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 328, in _invoke_callbacks
    callback(self)
  File "/home/op/env/lib/python3.8/site-packages/lithops/invokers.py", line 439, in _callback
    future.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/op/env/lib/python3.8/site-packages/lithops/invokers.py", line 370, in _invoke_task
    activation_id = self.compute_handler.invoke(payload)
  File "/home/op/env/lib/python3.8/site-packages/lithops/serverless/serverless.py", line 59, in invoke
    return self.backend.invoke(runtime_name, runtime_memory, job_payload)
  File "/home/op/env/lib/python3.8/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 577, in invoke
    raise Exception('Error {}: {}'.format(r.status_code, r.text))
Exception: Error 429: {"Reason":"CallerRateLimitExceeded","Type":"User","message":"Rate exceeded"}
exception calling callback for <Future at 0x7fed5f95ceb0 state=finished raised Exception>
Traceback (most recent call last):
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 328, in _invoke_callbacks
    callback(self)
  File "/home/op/env/lib/python3.8/site-packages/lithops/invokers.py", line 439, in _callback
    future.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/op/env/lib/python3.8/site-packages/lithops/invokers.py", line 370, in _invoke_task
    activation_id = self.compute_handler.invoke(payload)
  File "/home/op/env/lib/python3.8/site-packages/lithops/serverless/serverless.py", line 59, in invoke
    return self.backend.invoke(runtime_name, runtime_memory, job_payload)
  File "/home/op/env/lib/python3.8/site-packages/lithops/serverless/backends/aws_lambda/aws_lambda.py", line 577, in invoke
    raise Exception('Error {}: {}'.format(r.status_code, r.text))
Exception: Error 429: {"Reason":"CallerRateLimitExceeded","Type":"User","message":"Rate exceeded"}
exception calling callback for <Future at 0x7fedac1ae2b0 state=finished raised Exception>
fyoorer commented 1 year ago

This happens when your AWS account does not currently support more than 50 concurrent lambdas. this is common for new aws accounts, you can either reach to aws support to get your concurrent lambda executions count increased to 1000 or wait till it automatically increases after a few months