claffin / cloudproxy

Hide your scrapers IP behind the cloud. Provision proxy servers across different cloud providers to improve your scraping success.
https://cloudproxy.io/
MIT License
1.4k stars 80 forks source link

KeyError: 'PublicIpAddress' #24

Closed claffin closed 3 years ago

claffin commented 3 years ago

AWS creates the instance but doesn't allocate the IP instantly so when CloudProxy reads the response from AWS it cannot find the IP and causes the key error.

CloudProxy should still continue to work as once AWS allocates the IP, it stops raising the exception. Will fix in near future.

Traceback (most recent call last):

  File "/usr/local/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
    │    └ <function Thread._bootstrap_inner at 0x7fd7f2af6040>
    └ <Thread(ThreadPoolExecutor-0_1, started daemon 140565377611520)>
  File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
    │    └ <function Thread.run at 0x7fd7f2af5d30>
    └ <Thread(ThreadPoolExecutor-0_1, started daemon 140565377611520)>
  File "/usr/local/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <Thread(ThreadPoolExecutor-0_1, started daemon 140565377611520)>
    │    │        │    └ (<weakref at 0x7fd7efa8d400; to 'ThreadPoolExecutor' at 0x7fd7f12318b0>, <_queue.SimpleQueue object at 0x7fd7efb74950>, None,...
    │    │        └ <Thread(ThreadPoolExecutor-0_1, started daemon 140565377611520)>
    │    └ <function _worker at 0x7fd7f11f2d30>
    └ <Thread(ThreadPoolExecutor-0_1, started daemon 140565377611520)>
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 80, in _worker
    work_item.run()
    │         └ <function _WorkItem.run at 0x7fd7f11f2c10>
    └ <concurrent.futures.thread._WorkItem object at 0x7fd7efa93280>
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
             │    │   │    │       │    └ {}
             │    │   │    │       └ <concurrent.futures.thread._WorkItem object at 0x7fd7efa93280>
             │    │   │    └ [<Job (id=251677fd631043ddbb85e5197ee666dc name=aws_manager)>, 'default', [datetime.datetime(2021, 5, 10, 10, 34, 57, 579415,...
             │    │   └ <concurrent.futures.thread._WorkItem object at 0x7fd7efa93280>
             │    └ <function run_job at 0x7fd7f11fdf70>
             └ <concurrent.futures.thread._WorkItem object at 0x7fd7efa93280>
> File "/usr/local/lib/python3.8/site-packages/apscheduler/executors/base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
             │   │     │   │       │   └ <member 'kwargs' of 'Job' objects>
             │   │     │   │       └ <Job (id=251677fd631043ddbb85e5197ee666dc name=aws_manager)>
             │   │     │   └ <member 'args' of 'Job' objects>
             │   │     └ <Job (id=251677fd631043ddbb85e5197ee666dc name=aws_manager)>
             │   └ <member 'func' of 'Job' objects>
             └ <Job (id=251677fd631043ddbb85e5197ee666dc name=aws_manager)>

  File "/app/cloudproxy/providers/manager.py", line 15, in aws_manager
    ip_list = aws_start()
              └ <function aws_start at 0x7fd7efc91550>

  File "/app/cloudproxy/providers/aws/main.py", line 92, in aws_start
    aws_check_delete()
    └ <function aws_check_delete at 0x7fd7efc914c0>

  File "/app/cloudproxy/providers/aws/main.py", line 77, in aws_check_delete
    if instance["Instances"][0]["PublicIpAddress"] in delete_queue:
       │                                              └ set()
       └ {'Groups': [], 'Instances': [{'AmiLaunchIndex': 0, 'ImageId': 'ami-096cb92bb3580c759', 'InstanceId': 'i-0b18cc1721ffae51a', '...

KeyError: 'PublicIpAddress'

Originally posted by @sblfc in https://github.com/claffin/cloudproxy/issues/21#issuecomment-836534883