Open parasj opened 2 years ago
Describe the bug When running skyplane deprovision, I get the following error:
skyplane deprovision
skyplane deprovision ✘ 1 Deprovisioning 13 instances ⠋ Deprovisioning ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/13 0:00:0421:17:51 [ERROR] Error running <lambda>: An error occurred (NoSuchEntity) when calling the GetInstanceProfile operation: Instance Profile skyplane-aws-eaae092f_profile cannot be found. ⠹ Deprovisioning ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/13 0:00:0421:17:51 [ERROR] Error running <lambda>: An error occurred (NoSuchEntity) when calling the GetInstanceProfile operation: Instance Profile skyplane-aws-4b718763_profile cannot be found. 21:17:51 [ERROR] Error running <lambda>: An error occurred (NoSuchEntity) when calling the GetInstanceProfile operation: Instance Profile skyplane-aws-88dcb8d9_profile cannot be found. 21:17:51 [ERROR] Error running <lambda>: An error occurred (NoSuchEntity) when calling the GetInstanceProfile operation: Instance Profile skyplane-aws-fb09845d_profile cannot be found. 21:17:51 [ERROR] Error running <lambda>: An error occurred (NoSuchEntity) when calling the GetInstanceProfile operation: Instance Profile skyplane-aws-13c5f707_profile cannot be found. 21:17:51 [ERROR] Error running <lambda>: An error occurred (NoSuchEntity) when calling the GetInstanceProfile operation: Instance Profile skyplane-aws-e467d709_profile cannot be found. 21:17:51 [ERROR] Error running <lambda>: An error occurred (NoSuchEntity) when calling the GetInstanceProfile operation: Instance Profile skyplane-aws-8535d245_profile cannot be found. 21:17:51 [ERROR] Error running <lambda>: An error occurred (NoSuchEntity) when calling the GetInstanceProfile operation: Instance Profile skyplane-aws-bbd01006_profile cannot be found. 21:17:51 [ERROR] Error running <lambda>: An error occurred (NoSuchEntity) when calling the GetInstanceProfile operation: Instance Profile skyplane-aws-674378d1_profile cannot be found. 21:17:51 [ERROR] Error running <lambda>: An error occurred (NoSuchEntity) when calling the GetInstanceProfile operation: Instance Profile skyplane-aws-eff32554_profile cannot be found. 21:17:51 [ERROR] Error running <lambda>: An error occurred (NoSuchEntity) when calling the GetInstanceProfile operation: Instance Profile skyplane-aws-e2e9ad34_profile cannot be found. 21:17:51 [ERROR] Error running <lambda>: An error occurred (NoSuchEntity) when calling the GetInstanceProfile operation: Instance Profile skyplane-aws-d1395186_profile cannot be found. ⠦ Deprovisioning ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/13 0:00:0421:17:51 [ERROR] Error running <lambda>: An error occurred (NoSuchEntity) when calling the GetInstanceProfile operation: Instance Profile skyplane-aws-8ad031cf_profile cannot be found. ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ /Users/paras/code/skylark/skyplane/cli/cli.py:469 in deprovision │ │ │ │ 466 │ │ │ 467 │ if instances: │ │ 468 │ │ typer.secho(f"Deprovisioning {len(instances)} instances", fg="yellow", bold=True │ │ ❱ 469 │ │ do_parallel(lambda instance: instance.terminate_instance(), instances, desc="Dep │ │ 470 │ else: │ │ 471 │ │ typer.secho("No instances to deprovision", fg="yellow", bold=True) │ │ 472 │ │ │ │ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │ │ │ all = False │ │ │ │ filter_client_id = None │ │ │ │ instances = [ │ │ │ │ │ AWSServer(region_tag=aws:us-east-2, instance_id=i-00f31c80229d8a323), │ │ │ │ │ AWSServer(region_tag=aws:us-east-2, instance_id=i-003c6b1168fe19138), │ │ │ │ │ AWSServer(region_tag=aws:us-east-2, instance_id=i-03b4f7cf2ba8df60e), │ │ │ │ │ AWSServer(region_tag=aws:us-east-2, instance_id=i-0440d25860c3fc925), │ │ │ │ │ AWSServer(region_tag=aws:us-east-2, instance_id=i-05f404ad5438a685e), │ │ │ │ │ AWSServer(region_tag=aws:us-east-2, instance_id=i-03d0bd145c705cd74), │ │ │ │ │ AWSServer(region_tag=aws:us-east-2, instance_id=i-0645235e9de0a3886), │ │ │ │ │ AWSServer(region_tag=aws:us-east-1, instance_id=i-0f48a8f42c07aa9a4), │ │ │ │ │ AWSServer(region_tag=aws:us-east-1, instance_id=i-01419491bb9fa8d34), │ │ │ │ │ AWSServer(region_tag=aws:us-east-1, instance_id=i-05eb91c810efa9d47), │ │ │ │ │ ... +3 │ │ │ │ ] │ │ │ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /Users/paras/code/skylark/skyplane/utils/fn.py:57 in do_parallel │ │ │ │ 54 │ │ │ with ThreadPoolExecutor(max_workers=n) as executor: │ │ 55 │ │ │ │ future_list = [executor.submit(wrapped_fn, args) for args in args_list] │ │ 56 │ │ │ │ for future in as_completed(future_list): │ │ ❱ 57 │ │ │ │ │ args, result = future.result() │ │ 58 │ │ │ │ │ results.append((args, result)) │ │ 59 │ │ │ │ │ progress.update(progress_task, advance=1) │ │ 60 │ if spinner_persist: │ │ │ │ ╭────────────────────────────────────────── locals ───────────────────────────────────────────╮ │ │ │ arg_fmt = <function do_parallel.<locals>.<lambda> at 0x13dc98550> │ │ │ │ args_list = [ │ │ │ │ │ AWSServer(region_tag=aws:us-east-2, instance_id=i-00f31c80229d8a323), │ │ │ │ │ AWSServer(region_tag=aws:us-east-2, instance_id=i-003c6b1168fe19138), │ │ │ │ │ AWSServer(region_tag=aws:us-east-2, instance_id=i-03b4f7cf2ba8df60e), │ │ │ │ │ AWSServer(region_tag=aws:us-east-2, instance_id=i-0440d25860c3fc925), │ │ │ │ │ AWSServer(region_tag=aws:us-east-2, instance_id=i-05f404ad5438a685e), │ │ │ │ │ AWSServer(region_tag=aws:us-east-2, instance_id=i-03d0bd145c705cd74), │ │ │ │ │ AWSServer(region_tag=aws:us-east-2, instance_id=i-0645235e9de0a3886), │ │ │ │ │ AWSServer(region_tag=aws:us-east-1, instance_id=i-0f48a8f42c07aa9a4), │ │ │ │ │ AWSServer(region_tag=aws:us-east-1, instance_id=i-01419491bb9fa8d34), │ │ │ │ │ AWSServer(region_tag=aws:us-east-1, instance_id=i-05eb91c810efa9d47), │ │ │ │ │ ... +3 │ │ │ │ ] │ │ │ │ desc = 'Deprovisioning' │ │ │ │ executor = <concurrent.futures.thread.ThreadPoolExecutor object at 0x168d5cc10> │ │ │ │ func = <function deprovision.<locals>.<lambda> at 0x13dc985e0> │ │ │ │ future = <Future at 0x294928f70 state=finished raised NoSuchEntityException> │ │ │ │ future_list = [ │ │ │ │ │ <Future at 0x168d5c820 state=finished raised NoSuchEntityException>, │ │ │ │ │ <Future at 0x168d5cb20 state=finished raised NoSuchEntityException>, │ │ │ │ │ <Future at 0x290cfbb20 state=finished raised NoSuchEntityException>, │ │ │ │ │ <Future at 0x295712920 state=finished raised NoSuchEntityException>, │ │ │ │ │ <Future at 0x294928f70 state=finished raised NoSuchEntityException>, │ │ │ │ │ <Future at 0x2948a7fd0 state=finished raised NoSuchEntityException>, │ │ │ │ │ <Future at 0x291e4c610 state=finished raised NoSuchEntityException>, │ │ │ │ │ <Future at 0x29029c190 state=finished raised NoSuchEntityException>, │ │ │ │ │ <Future at 0x290d86560 state=finished raised NoSuchEntityException>, │ │ │ │ │ <Future at 0x290d241f0 state=finished raised NoSuchEntityException>, │ │ │ │ │ ... +3 │ │ │ │ ] │ │ │ │ n = 13 │ │ │ │ progress = <rich.progress.Progress object at 0x291f1b0a0> │ │ │ │ progress_task = 0 │ │ │ │ results = [] │ │ │ │ return_args = True │ │ │ │ spinner = True │ │ │ │ spinner_persist = True │ │ │ │ t = <skyplane.utils.timer.Timer object at 0x168d5dc30> │ │ │ │ wrapped_fn = <function do_parallel.<locals>.wrapped_fn at 0x13dc98700> │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /opt/homebrew/Caskroom/miniconda/base/lib/python3.10/concurrent/futures/_base.py:451 in result │ │ │ │ 448 │ │ │ │ if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]: │ │ 449 │ │ │ │ │ raise CancelledError() │ │ 450 │ │ │ │ elif self._state == FINISHED: │ │ ❱ 451 │ │ │ │ │ return self.__get_result() │ │ 452 │ │ │ │ │ │ 453 │ │ │ │ self._condition.wait(timeout) │ │ 454 │ │ │ │ ╭──── locals ────╮ │ │ │ self = None │ │ │ │ timeout = None │ │ │ ╰────────────────╯ │ │ │ │ /opt/homebrew/Caskroom/miniconda/base/lib/python3.10/concurrent/futures/_base.py:403 in │ │ __get_result │ │ │ │ 400 │ def __get_result(self): │ │ 401 │ │ if self._exception: │ │ 402 │ │ │ try: │ │ ❱ 403 │ │ │ │ raise self._exception │ │ 404 │ │ │ finally: │ │ 405 │ │ │ │ # Break a reference cycle with the exception in self._exception │ │ 406 │ │ │ │ self = None │ │ │ │ ╭── locals ───╮ │ │ │ self = None │ │ │ ╰─────────────╯ │ │ │ │ /opt/homebrew/Caskroom/miniconda/base/lib/python3.10/concurrent/futures/thread.py:58 in run │ │ │ │ 55 │ │ │ return │ │ 56 │ │ │ │ 57 │ │ try: │ │ ❱ 58 │ │ │ result = self.fn(*self.args, **self.kwargs) │ │ 59 │ │ except BaseException as exc: │ │ 60 │ │ │ self.future.set_exception(exc) │ │ 61 │ │ │ # Break a reference cycle with the exception 'exc' │ │ │ │ ╭── locals ───╮ │ │ │ self = None │ │ │ ╰─────────────╯ │ │ │ │ /Users/paras/code/skylark/skyplane/utils/fn.py:43 in wrapped_fn │ │ │ │ 40 │ │ │ 41 │ def wrapped_fn(args): │ │ 42 │ │ try: │ │ ❱ 43 │ │ │ return args, func(args) │ │ 44 │ │ except Exception as e: │ │ 45 │ │ │ logger.error(f"Error running {func.__name__}: {e}") │ │ 46 │ │ │ raise │ │ │ │ ╭────────────────────────────────── locals ───────────────────────────────────╮ │ │ │ args = AWSServer(region_tag=aws:us-east-2, instance_id=i-05f404ad5438a685e) │ │ │ │ func = <function deprovision.<locals>.<lambda> at 0x13dc985e0> │ │ │ ╰─────────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /Users/paras/code/skylark/skyplane/cli/cli.py:469 in <lambda> │ │ │ │ 466 │ │ │ 467 │ if instances: │ │ 468 │ │ typer.secho(f"Deprovisioning {len(instances)} instances", fg="yellow", bold=True │ │ ❱ 469 │ │ do_parallel(lambda instance: instance.terminate_instance(), instances, desc="Dep │ │ 470 │ else: │ │ 471 │ │ typer.secho("No instances to deprovision", fg="yellow", bold=True) │ │ 472 │ │ │ │ ╭──────────────────────────────────── locals ─────────────────────────────────────╮ │ │ │ instance = AWSServer(region_tag=aws:us-east-2, instance_id=i-05f404ad5438a685e) │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /Users/paras/code/skylark/skyplane/compute/server.py:169 in terminate_instance │ │ │ │ 166 │ def terminate_instance(self): │ │ 167 │ │ """Terminate instance""" │ │ 168 │ │ self.close_server() │ │ ❱ 169 │ │ self.terminate_instance_impl() │ │ 170 │ │ │ 171 │ def enable_auto_shutdown(self, timeout_minutes=None): │ │ 172 │ │ if timeout_minutes is None: │ │ │ │ ╭────────────────────────────────── locals ───────────────────────────────────╮ │ │ │ self = AWSServer(region_tag=aws:us-east-2, instance_id=i-05f404ad5438a685e) │ │ │ ╰─────────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /Users/paras/code/skylark/skyplane/compute/aws/aws_server.py:112 in terminate_instance_impl │ │ │ │ 109 │ │ │ profile = iam.InstanceProfile(profile["Arn"].split("/")[-1]) │ │ 110 │ │ │ │ │ 111 │ │ │ # remove all roles from instance profile │ │ ❱ 112 │ │ │ for role in profile.roles: │ │ 113 │ │ │ │ profile.remove_role(RoleName=role.name) │ │ 114 │ │ │ │ │ 115 │ │ │ # delete instance profile │ │ │ │ ╭──────────────────────────────────── locals ────────────────────────────────────╮ │ │ │ iam = iam.ServiceResource() │ │ │ │ profile = iam.InstanceProfile(name='skyplane-aws-eaae092f_profile') │ │ │ │ self = AWSServer(region_tag=aws:us-east-2, instance_id=i-05f404ad5438a685e) │ │ │ ╰────────────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /opt/homebrew/Caskroom/miniconda/base/lib/python3.10/site-packages/boto3/resources/factory.py:48 │ │ 5 in get_reference │ │ │ │ 482 │ │ │ # the handler as if it were a response. This allows references │ │ 483 │ │ │ # to have their data loaded properly. │ │ 484 │ │ │ if needs_data and self.meta.data is None and hasattr(self, 'load'): │ │ ❱ 485 │ │ │ │ self.load() │ │ 486 │ │ │ return handler(self, {}, self.meta.data) │ │ 487 │ │ │ │ 488 │ │ get_reference.__name__ = str(reference_model.name) │ │ │ │ ╭─────────────────────────────────── locals ────────────────────────────────────╮ │ │ │ handler = <boto3.resources.response.ResourceHandler object at 0x17e26e7d0> │ │ │ │ needs_data = True │ │ │ │ self = iam.InstanceProfile(name='skyplane-aws-eaae092f_profile') │ │ │ ╰───────────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /opt/homebrew/Caskroom/miniconda/base/lib/python3.10/site-packages/boto3/resources/factory.py:56 │ │ 4 in do_action │ │ │ │ 561 │ │ │ # We need a new method here because we want access to the │ │ 562 │ │ │ # instance via ``self``. │ │ 563 │ │ │ def do_action(self, *args, **kwargs): │ │ ❱ 564 │ │ │ │ response = action(self, *args, **kwargs) │ │ 565 │ │ │ │ self.meta.data = response │ │ 566 │ │ │ │ │ 567 │ │ │ # Create the docstring for the load/reload methods. │ │ │ │ ╭─────────────────────────────── locals ────────────────────────────────╮ │ │ │ action = <boto3.resources.action.ServiceAction object at 0x17e2166e0> │ │ │ │ args = () │ │ │ │ kwargs = {} │ │ │ │ self = iam.InstanceProfile(name='skyplane-aws-eaae092f_profile') │ │ │ ╰───────────────────────────────────────────────────────────────────────╯ │ │ │ │ /opt/homebrew/Caskroom/miniconda/base/lib/python3.10/site-packages/boto3/resources/action.py:88 │ │ in __call__ │ │ │ │ 85 │ │ │ params, │ │ 86 │ │ ) │ │ 87 │ │ │ │ ❱ 88 │ │ response = getattr(parent.meta.client, operation_name)(*args, **params) │ │ 89 │ │ │ │ 90 │ │ logger.debug('Response: %r', response) │ │ 91 │ │ │ │ ╭─────────────────────────────────── locals ────────────────────────────────────╮ │ │ │ args = () │ │ │ │ kwargs = {} │ │ │ │ operation_name = 'get_instance_profile' │ │ │ │ params = {'InstanceProfileName': 'skyplane-aws-eaae092f_profile'} │ │ │ │ parent = iam.InstanceProfile(name='skyplane-aws-eaae092f_profile') │ │ │ │ self = <boto3.resources.action.ServiceAction object at 0x17e2166e0> │ │ │ ╰───────────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /opt/homebrew/Caskroom/miniconda/base/lib/python3.10/site-packages/botocore/client.py:515 in │ │ _api_call │ │ │ │ 512 │ │ │ │ │ f"{py_operation_name}() only accepts keyword arguments." │ │ 513 │ │ │ │ ) │ │ 514 │ │ │ # The "self" in this scope is referring to the BaseClient. │ │ ❱ 515 │ │ │ return self._make_api_call(operation_name, kwargs) │ │ 516 │ │ │ │ 517 │ │ _api_call.__name__ = str(py_operation_name) │ │ 518 │ │ │ │ ╭─────────────────────────────────── locals ───────────────────────────────────╮ │ │ │ args = () │ │ │ │ kwargs = {'InstanceProfileName': 'skyplane-aws-eaae092f_profile'} │ │ │ │ operation_name = 'GetInstanceProfile' │ │ │ │ py_operation_name = 'get_instance_profile' │ │ │ │ self = <botocore.client.IAM object at 0x291359b40> │ │ │ ╰──────────────────────────────────────────────────────────────────────────────╯ │ │ │ │ /opt/homebrew/Caskroom/miniconda/base/lib/python3.10/site-packages/botocore/client.py:934 in │ │ _make_api_call │ │ │ │ 931 │ │ if http.status_code >= 300: │ │ 932 │ │ │ error_code = parsed_response.get("Error", {}).get("Code") │ │ 933 │ │ │ error_class = self.exceptions.from_code(error_code) │ │ ❱ 934 │ │ │ raise error_class(parsed_response, operation_name) │ │ 935 │ │ else: │ │ 936 │ │ │ return parsed_response │ │ 937 │ │ │ │ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │ │ │ additional_headers = {} │ │ │ │ api_params = {'InstanceProfileName': 'skyplane-aws-eaae092f_profile'} │ │ │ │ endpoint_url = 'https://iam.amazonaws.com' │ │ │ │ error_class = <class 'botocore.errorfactory.NoSuchEntityException'> │ │ │ │ error_code = 'NoSuchEntity' │ │ │ │ event_response = None │ │ │ │ handler = <function inject_api_version_header_if_needed at 0x13e123b50> │ │ │ │ http = <botocore.awsrequest.AWSResponse object at 0x17f0c0340> │ │ │ │ operation_model = OperationModel(name=GetInstanceProfile) │ │ │ │ operation_name = 'GetInstanceProfile' │ │ │ │ parsed_response = { │ │ │ │ │ 'Error': { │ │ │ │ │ │ 'Type': 'Sender', │ │ │ │ │ │ 'Code': 'NoSuchEntity', │ │ │ │ │ │ 'Message': 'Instance Profile skyplane-aws-eaae092f_profile │ │ │ │ cannot be found.' │ │ │ │ │ }, │ │ │ │ │ 'ResponseMetadata': { │ │ │ │ │ │ 'RequestId': '7f4c5574-4b56-4d76-a34b-80a8cfb1bf6f', │ │ │ │ │ │ 'HTTPStatusCode': 404, │ │ │ │ │ │ 'HTTPHeaders': { │ │ │ │ │ │ │ 'x-amzn-requestid': '7f4c5574-4b56-4d76-a34b-80a8cfb1bf6f', │ │ │ │ │ │ │ 'content-type': 'text/xml', │ │ │ │ │ │ │ 'content-length': '307', │ │ │ │ │ │ │ 'date': 'Tue, 22 Nov 2022 05:17:50 GMT' │ │ │ │ │ │ }, │ │ │ │ │ │ 'RetryAttempts': 0 │ │ │ │ │ } │ │ │ │ } │ │ │ │ request_context = { │ │ │ │ │ 'client_region': 'aws-global', │ │ │ │ │ 'client_config': <botocore.config.Config object at 0x29135b310>, │ │ │ │ │ 'has_streaming_input': False, │ │ │ │ │ 'auth_type': 'v4', │ │ │ │ │ 'signing': {'region': 'us-east-1', 'signing_name': 'iam'}, │ │ │ │ │ 'retries': { │ │ │ │ │ │ 'attempt': 1, │ │ │ │ │ │ 'invocation-id': '70812e64-ff35-47d4-936e-3e0dff558973', │ │ │ │ │ │ 'max': 5 │ │ │ │ │ }, │ │ │ │ │ 'timestamp': '20221122T051750Z' │ │ │ │ } │ │ │ │ request_dict = { │ │ │ │ │ 'url_path': '/', │ │ │ │ │ 'query_string': '', │ │ │ │ │ 'method': 'POST', │ │ │ │ │ 'headers': { │ │ │ │ │ │ 'Content-Type': 'application/x-www-form-urlencoded; │ │ │ │ charset=utf-8', │ │ │ │ │ │ 'User-Agent': 'Boto3/1.26.3 Python/3.10.6 Darwin/22.1.0 │ │ │ │ Botocore/1.29.9 Resource' │ │ │ │ │ }, │ │ │ │ │ 'body': { │ │ │ │ │ │ 'Action': 'GetInstanceProfile', │ │ │ │ │ │ 'Version': '2010-05-08', │ │ │ │ │ │ 'InstanceProfileName': 'skyplane-aws-eaae092f_profile' │ │ │ │ │ }, │ │ │ │ │ 'url': 'https://iam.amazonaws.com/', │ │ │ │ │ 'context': { │ │ │ │ │ │ 'client_region': 'aws-global', │ │ │ │ │ │ 'client_config': <botocore.config.Config object at │ │ │ │ 0x29135b310>, │ │ │ │ │ │ 'has_streaming_input': False, │ │ │ │ │ │ 'auth_type': 'v4', │ │ │ │ │ │ 'signing': {'region': 'us-east-1', 'signing_name': 'iam'}, │ │ │ │ │ │ 'retries': { │ │ │ │ │ │ │ 'attempt': 1, │ │ │ │ │ │ │ 'invocation-id': '70812e64-ff35-47d4-936e-3e0dff558973', │ │ │ │ │ │ │ 'max': 5 │ │ │ │ │ │ }, │ │ │ │ │ │ 'timestamp': '20221122T051750Z' │ │ │ │ │ } │ │ │ │ } │ │ │ │ self = <botocore.client.IAM object at 0x291359b40> │ │ │ │ service_id = 'iam' │ │ │ │ service_name = 'iam' │ │ │ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ NoSuchEntityException: An error occurred (NoSuchEntity) when calling the GetInstanceProfile operation: Instance Profile skyplane-aws-eaae092f_profile cannot be found. (base) ~/c/skylark ❯❯❯
Environment info (please complete the following information):
Is this still an issue?
sarahwooders
commented
1 year ago sarahwooders
commented
1 year ago
Describe the bug When running
skyplane deprovision, I get the following error:Environment info (please complete the following information):