search

nebari-dev / nebari

🪴 Nebari - your open source data science platform
https://nebari.dev
BSD 3-Clause "New" or "Revised" License
279 stars 89 forks source link

[BUG] - KeyError: 'realm_id' during destroy #1814

Open pmeier opened 1 year ago

pmeier commented 1 year ago

Describe the bug

KeyError: 'realm_id'

while running nebari destroy. Full traceback below.

Expected behavior

Destroy cleanly.

OS and architecture in which you are running Nebari

Arch x64

How to Reproduce the problem?

I tried a deployment to AWS and aborted at the DNS setup. Afterwards I ran nebari destroy and was met with a KeyError.

Command output

❯ nebari destroy -c nebari-config.yaml
Are you sure you want to destroy your Nebari cluster? [y/N]: y
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ The following files will be updated:            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ stages/03-kubernetes-initialize/_nebari.tf.json │
│ stages/04-kubernetes-ingress/_nebari.tf.json    │
│ stages/05-kubernetes-keycloak/_nebari.tf.json   │
│ stages/07-kubernetes-services/_nebari.tf.json   │
│ stages/08-nebari-tf-extensions/_nebari.tf.json  │
└─────────────────────────────────────────────────┘
[terraform]: Upgrading modules...
[terraform]: - terraform-state in modules/terraform-state
[terraform]: 
[terraform]: Initializing the backend...
[terraform]: 
[terraform]: Initializing provider plugins...
[terraform]: - Finding hashicorp/aws versions matching "3.73.0"...
[terraform]: - Using previously-installed hashicorp/aws v3.73.0
[terraform]: 
[terraform]: Terraform has been successfully initialized!
[terraform]: 
[terraform]: You may now begin working with Terraform. Try running "terraform plan" to see
[terraform]: any changes that are required for your infrastructure. All Terraform commands
[terraform]: should now work.
[terraform]: 
[terraform]: If you ever set or change modules or backend configuration for Terraform,
[terraform]: rerun this command to reinitialize your working directory. If you forget, other
[terraform]: commands will detect it and remind you to do so if necessary.
['import', '-var-file=/tmp/tmp9hmnoy5b.tfvars.json', 'module.terraform-state.aws_s3_bucket.terraform-state', 'pmeier-aws-test-dev-terraform-state']
[terraform]: module.terraform-state.aws_s3_bucket.terraform-state: Importing from ID "pmeier-aws-test-dev-terraform-state"...
[terraform]: module.terraform-state.aws_s3_bucket.terraform-state: Import prepared!
[terraform]:   Prepared aws_s3_bucket for import
[terraform]: ╷
[terraform]: │ Error: Resource already managed by Terraform
[terraform]: │ 
[terraform]: │ Terraform is already managing a remote object for
[terraform]: │ module.terraform-state.aws_s3_bucket.terraform-state. To import to this
[terraform]: │ address you must first remove the existing object from the state.
[terraform]: ╵
[terraform]: 
['import', '-var-file=/tmp/tmp9hmnoy5b.tfvars.json', 'module.terraform-state.aws_dynamodb_table.terraform-state-lock', 'pmeier-aws-test-dev-terraform-state-lock']
[terraform]: module.terraform-state.aws_dynamodb_table.terraform-state-lock: Importing from ID "pmeier-aws-test-dev-terraform-state-lock"...
[terraform]: module.terraform-state.aws_dynamodb_table.terraform-state-lock: Import prepared!
[terraform]:   Prepared aws_dynamodb_table for import
[terraform]: ╷
[terraform]: │ Error: Resource already managed by Terraform
[terraform]: │ 
[terraform]: │ Terraform is already managing a remote object for
[terraform]: │ module.terraform-state.aws_dynamodb_table.terraform-state-lock. To import
[terraform]: │ to this address you must first remove the existing object from the state.
[terraform]: ╵
[terraform]: 
[terraform]: Upgrading modules...
[terraform]: - accounting in modules/accounting
[terraform]: - efs in modules/efs
[terraform]: - kubernetes in modules/kubernetes
[terraform]: - network in modules/network
[terraform]: - registry-jupyterlab in modules/registry
[terraform]: 
[terraform]: Initializing the backend...
[terraform]: 
[terraform]: Initializing provider plugins...
[terraform]: - Finding hashicorp/aws versions matching "3.73.0"...
[terraform]: - Finding latest version of hashicorp/local...
[terraform]: - Using previously-installed hashicorp/aws v3.73.0
[terraform]: - Using previously-installed hashicorp/local v2.4.0
[terraform]: 
[terraform]: Terraform has been successfully initialized!
[terraform]: 
[terraform]: You may now begin working with Terraform. Try running "terraform plan" to see
[terraform]: any changes that are required for your infrastructure. All Terraform commands
[terraform]: should now work.
[terraform]: 
[terraform]: If you ever set or change modules or backend configuration for Terraform,
[terraform]: rerun this command to reinitialize your working directory. If you forget, other
[terraform]: commands will detect it and remind you to do so if necessary.
[terraform]: Upgrading modules...
[terraform]: - external-container-reg in modules/extcr
[terraform]: - kubernetes-autoscaling in modules/cluster-autoscaler
[terraform]: - kubernetes-initialization in modules/initialization
[terraform]: - nvidia-driver-installer in modules/nvidia-installer
[terraform]: - traefik-crds in modules/traefik_crds
[terraform]: 
[terraform]: Initializing the backend...
[terraform]: 
[terraform]: Initializing provider plugins...
[terraform]: - Finding hashicorp/kubernetes versions matching "2.20.0"...
[terraform]: - Finding hashicorp/helm versions matching "2.1.2"...
[terraform]: - Finding latest version of hashicorp/aws...
[terraform]: - Using previously-installed hashicorp/kubernetes v2.20.0
[terraform]: - Using previously-installed hashicorp/helm v2.1.2
[terraform]: - Using previously-installed hashicorp/aws v4.67.0
[terraform]: 
[terraform]: Terraform has been successfully initialized!
[terraform]: 
[terraform]: You may now begin working with Terraform. Try running "terraform plan" to see
[terraform]: any changes that are required for your infrastructure. All Terraform commands
[terraform]: should now work.
[terraform]: 
[terraform]: If you ever set or change modules or backend configuration for Terraform,
[terraform]: rerun this command to reinitialize your working directory. If you forget, other
[terraform]: commands will detect it and remind you to do so if necessary.
[terraform]: Upgrading modules...
[terraform]: - kubernetes-ingress in modules/kubernetes/ingress
[terraform]: 
[terraform]: Initializing the backend...
[terraform]: 
[terraform]: Initializing provider plugins...
[terraform]: - Finding hashicorp/helm versions matching "2.1.2"...
[terraform]: - Finding hashicorp/kubernetes versions matching "2.20.0"...
[terraform]: - Finding latest version of hashicorp/aws...
[terraform]: - Using previously-installed hashicorp/helm v2.1.2
[terraform]: - Using previously-installed hashicorp/kubernetes v2.20.0
[terraform]: - Using previously-installed hashicorp/aws v4.67.0
[terraform]: 
[terraform]: Terraform has been successfully initialized!
[terraform]: 
[terraform]: You may now begin working with Terraform. Try running "terraform plan" to see
[terraform]: any changes that are required for your infrastructure. All Terraform commands
[terraform]: should now work.
[terraform]: 
[terraform]: If you ever set or change modules or backend configuration for Terraform,
[terraform]: rerun this command to reinitialize your working directory. If you forget, other
[terraform]: commands will detect it and remind you to do so if necessary.
[terraform]: Upgrading modules...
[terraform]: - kubernetes-keycloak-helm in modules/kubernetes/keycloak-helm
[terraform]: 
[terraform]: Initializing the backend...
[terraform]: 
[terraform]: Successfully configured the backend "s3"! Terraform will automatically
[terraform]: use this backend unless the backend configuration changes.
[terraform]: 
[terraform]: Initializing provider plugins...
[terraform]: - Finding hashicorp/helm versions matching "2.1.2"...
[terraform]: - Finding hashicorp/kubernetes versions matching "2.20.0"...
[terraform]: - Finding latest version of hashicorp/random...
[terraform]: - Finding latest version of hashicorp/aws...
[terraform]: - Installing hashicorp/helm v2.1.2...
[terraform]: - Installed hashicorp/helm v2.1.2 (signed by HashiCorp)
[terraform]: - Installing hashicorp/kubernetes v2.20.0...
[terraform]: - Installed hashicorp/kubernetes v2.20.0 (signed by HashiCorp)
[terraform]: - Installing hashicorp/random v3.5.1...
[terraform]: - Installed hashicorp/random v3.5.1 (signed by HashiCorp)
[terraform]: - Installing hashicorp/aws v4.67.0...
[terraform]: - Installed hashicorp/aws v4.67.0 (signed by HashiCorp)
[terraform]: 
[terraform]: Terraform has created a lock file .terraform.lock.hcl to record the provider
[terraform]: selections it made above. Include this file in your version control repository
[terraform]: so that Terraform can guarantee to make the same selections by default when
[terraform]: you run "terraform init" in the future.
[terraform]: 
[terraform]: Terraform has been successfully initialized!
[terraform]: 
[terraform]: You may now begin working with Terraform. Try running "terraform plan" to see
[terraform]: any changes that are required for your infrastructure. All Terraform commands
[terraform]: should now work.
[terraform]: 
[terraform]: If you ever set or change modules or backend configuration for Terraform,
[terraform]: rerun this command to reinitialize your working directory. If you forget, other
[terraform]: commands will detect it and remind you to do so if necessary.
[terraform]: 
[terraform]: Initializing the backend...
[terraform]: 
[terraform]: Successfully configured the backend "s3"! Terraform will automatically
[terraform]: use this backend unless the backend configuration changes.
[terraform]: 
[terraform]: Initializing provider plugins...
[terraform]: - Finding hashicorp/helm versions matching "2.1.2"...
[terraform]: - Finding hashicorp/kubernetes versions matching "2.20.0"...
[terraform]: - Finding mrparkers/keycloak versions matching "3.7.0"...
[terraform]: - Installing hashicorp/helm v2.1.2...
[terraform]: - Installed hashicorp/helm v2.1.2 (signed by HashiCorp)
[terraform]: - Installing hashicorp/kubernetes v2.20.0...
[terraform]: - Installed hashicorp/kubernetes v2.20.0 (signed by HashiCorp)
[terraform]: - Installing mrparkers/keycloak v3.7.0...
[terraform]: - Installed mrparkers/keycloak v3.7.0 (self-signed, key ID C50867915E116CD2)
[terraform]: 
[terraform]: Partner and community providers are signed by their developers.
[terraform]: If you'd like to know more about provider signing, you can read about it here:
[terraform]: https://www.terraform.io/docs/cli/plugins/signing.html
[terraform]: 
[terraform]: Terraform has created a lock file .terraform.lock.hcl to record the provider
[terraform]: selections it made above. Include this file in your version control repository
[terraform]: so that Terraform can guarantee to make the same selections by default when
[terraform]: you run "terraform init" in the future.
[terraform]: 
[terraform]: Terraform has been successfully initialized!
[terraform]: 
[terraform]: You may now begin working with Terraform. Try running "terraform plan" to see
[terraform]: any changes that are required for your infrastructure. All Terraform commands
[terraform]: should now work.
[terraform]: 
[terraform]: If you ever set or change modules or backend configuration for Terraform,
[terraform]: rerun this command to reinitialize your working directory. If you forget, other
[terraform]: commands will detect it and remind you to do so if necessary.
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/philip/git/nebari-dev/nebari/nebari/cli/main.py:379 in destroy                             │
│                                                                                                  │
│   376 │   if disable_prompt:                                                                     │
│   377 │   │   _run_destroy()                                                                     │
│   378 │   elif typer.confirm("Are you sure you want to destroy your Nebari cluster?"):           │
│ ❱ 379 │   │   _run_destroy()                                                                     │
│   380 │   else:                                                                                  │
│   381 │   │   raise typer.Abort()                                                                │
│   382                                                                                            │
│                                                                                                  │
│ ╭────────────────────────────────── locals ───────────────────────────────────╮                  │
│ │   _run_destroy = <function destroy.<locals>._run_destroy at 0x7f4246461a60> │                  │
│ │         config = 'nebari-config.yaml'                                       │                  │
│ │ disable_prompt = False                                                      │                  │
│ │ disable_render = False                                                      │                  │
│ │         output = './'                                                       │                  │
│ ╰─────────────────────────────────────────────────────────────────────────────╯                  │
│                                                                                                  │
│ /home/philip/git/nebari-dev/nebari/nebari/cli/main.py:374 in _run_destroy                        │
│                                                                                                  │
│   371 │   │   if not disable_render:                                                             │
│   372 │   │   │   render_template(output, config, force=True)                                    │
│   373 │   │                                                                                      │
│ ❱ 374 │   │   destroy_configuration(config_yaml)                                                 │
│   375 │                                                                                          │
│   376 │   if disable_prompt:                                                                     │
│   377 │   │   _run_destroy()                                                                     │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │          config = 'nebari-config.yaml'                                                       │ │
│ │ config_filename = PosixPath('nebari-config.yaml')                                            │ │
│ │     config_yaml = ordereddict([('project_name', 'pmeier-aws-test'), ('provider', 'aws'),     │ │
│ │                   ('domain', 'nebari.pystiche.org'), ('certificate', ordereddict([('type',   │ │
│ │                   'lets-encrypt'), ('acme_email', 'pmeier@quansight.com'), ('acme_server',   │ │
│ │                   'https://acme-v02.api.letsencrypt.org/directory')])), ('security',         │ │
│ │                   ordereddict([('authentication', ordereddict([('type', 'password')])),      │ │
│ │                   ('keycloak', ordereddict([('initial_root_password',                        │ │
│ │                   'Nv4T6wwrvXt9KmEF')]))])), ('default_images', ordereddict([('jupyterhub',  │ │
│ │                   'quay.io/nebari/nebari-jupyterhub:2023.5.1'), ('jupyterlab',               │ │
│ │                   'quay.io/nebari/nebari-jupyterlab:2023.5.1'), ('dask_worker',              │ │
│ │                   'quay.io/nebari/nebari-dask-worker:2023.5.1')])), ('storage',              │ │
│ │                   ordereddict([('conda_store', '200Gi'), ('shared_filesystem', '200Gi')])),  │ │
│ │                   ('theme', ordereddict([('jupyterhub', ordereddict([('hub_title', 'Nebari - │ │
│ │                   pmeier-aws-test'), ('hub_subtitle', 'Your open source data science         │ │
│ │                   platform, hosted on Amazon Web Services'), ('welcome', 'Welcome! Learn     │ │
│ │                   about Nebari\'s features and configurations in <a                          │ │
│ │                   href="https://www.nebari.dev/docs">the documentation</a>. If you have any  │ │
│ │                   questions or feedback, reach the team on <a                                │ │
│ │                   href="https://www.nebari.dev/docs/community#getting-support">Nebari\'s     │ │
│ │                   support forums</a>.'), ('logo',                                            │ │
│ │                   'https://raw.githubusercontent.com/nebari-dev/nebari-design/main/logo-mar… │ │
│ │                   ('display_version', True), ('version', 'v2023.5.2.dev2+g8bd56f90')]))])),  │ │
│ │                   ('helm_extensions', []), ('monitoring', ordereddict([('enabled', True)])), │ │
│ │                   ('argo_workflows', ordereddict([('enabled', True)])), ('kbatch',           │ │
│ │                   ordereddict([('enabled', True)])), ('cdsdashboards',                       │ │
│ │                   ordereddict([('enabled', True), ('cds_hide_user_named_servers', True),     │ │
│ │                   ('cds_hide_user_dashboard_servers', False)])), ('terraform_state',         │ │
│ │                   ordereddict([('type', 'remote')])), ('namespace', 'dev'),                  │ │
│ │                   ('nebari_version', '2023.5.2.dev2+g8bd56f90'), ('amazon_web_services',     │ │
│ │                   ordereddict([('region', 'us-west-2'), ('kubernetes_version', '1.24'),      │ │
│ │                   ('node_groups', ordereddict([('general', ordereddict([('instance',         │ │
│ │                   'm5.2xlarge'), ('min_nodes', 1), ('max_nodes', 1)])), ('user',             │ │
│ │                   ordereddict([('instance', 'm5.xlarge'), ('min_nodes', 1), ('max_nodes',    │ │
│ │                   5), ('single_subnet', False)])), ('worker', ordereddict([('instance',      │ │
│ │                   'm5.xlarge'), ('min_nodes', 1), ('max_nodes', 5), ('single_subnet',        │ │
│ │                   False)]))]))])), ('profiles', ordereddict([('jupyterlab',                  │ │
│ │                   [ordereddict([('display_name', 'Small Instance'), ('description', 'Stable  │ │
│ │                   environment with 2 cpu / 8 GB ram'), ('default', True),                    │ │
│ │                   ('kubespawner_override', ordereddict([('cpu_limit', 2), ('cpu_guarantee',  │ │
│ │                   1.5), ('mem_limit', '8G'), ('mem_guarantee', '5G')]))]),                   │ │
│ │                   ordereddict([('display_name', 'Medium Instance'), ('description', 'Stable  │ │
│ │                   environment with 4 cpu / 16 GB ram'), ('kubespawner_override',             │ │
│ │                   ordereddict([('cpu_limit', 4), ('cpu_guarantee', 3), ('mem_limit', '16G'), │ │
│ │                   ('mem_guarantee', '10G')]))])]), ('dask_worker', ordereddict([('Small      │ │
│ │                   Worker', ordereddict([('worker_cores_limit', 2), ('worker_cores', 1.5),    │ │
│ │                   ('worker_memory_limit', '8G'), ('worker_memory', '5G'), ('worker_threads', │ │
│ │                   2)])), ('Medium Worker', ordereddict([('worker_cores_limit', 4),           │ │
│ │                   ('worker_cores', 3), ('worker_memory_limit', '16G'), ('worker_memory',     │ │
│ │                   '10G'), ('worker_threads', 4)]))]))])), ('environments',                   │ │
│ │                   ordereddict([('environment-dask.yaml', ordereddict([('name', 'dask'),      │ │
│ │                   ('channels', ['conda-forge']), ('dependencies', ['python=3.10.8',          │ │
│ │                   'ipykernel=6.21.0', 'ipywidgets==7.7.1', 'nebari-dask ==2023.5.1',         │ │
│ │                   'python-graphviz=0.20.1', 'pyarrow=10.0.1', 's3fs=2023.1.0',               │ │
│ │                   'gcsfs=2023.1.0', 'numpy=1.23.5', 'numba=0.56.4', 'pandas=1.5.3',          │ │
│ │                   ordereddict([('pip', ['kbatch==0.4.1'])])])])),                            │ │
│ │                   ('environment-dashboard.yaml', ordereddict([('name', 'dashboard'),         │ │
│ │                   ('channels', ['conda-forge']), ('dependencies', ['python=3.10',            │ │
│ │                   'cdsdashboards-singleuser=0.6.3', 'cufflinks-py=0.17.3', 'dash=2.8.1',     │ │
│ │                   'geopandas=0.12.2', 'geopy=2.3.0', 'geoviews=1.9.6', 'gunicorn=20.1.0',    │ │
│ │                   'holoviews=1.15.4', 'ipykernel=6.21.2', 'ipywidgets=8.0.4',                │ │
│ │                   'jupyter=1.0.0', 'jupyterlab=3.6.1', 'jupyter_bokeh=3.0.5',                │ │
│ │                   'matplotlib=3.7.0', 'nebari-dask==2023.5.1', 'nodejs=18.12.1', 'numpy',    │ │
│ │                   'openpyxl=3.1.1', 'pandas=1.5.3', 'panel=0.14.3', 'param=1.12.3',          │ │
│ │                   'plotly=5.13.0', 'python-graphviz=0.20.1', 'rich=13.3.1',                  │ │
│ │                   'streamlit=1.9.0', 'sympy=1.11.1', 'voila=0.4.0', 'pip=23.0',              │ │
│ │                   ordereddict([('pip', ['streamlit-image-comparison==0.0.3',                 │ │
│ │                   'noaa-coops==0.2.1', 'dash_core_components==2.0.0',                        │ │
│ │                   'dash_html_components==2.0.0'])])])]))]))])                                │ │
│ │  disable_render = False                                                                      │ │
│ │          output = './'                                                                       │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│ /home/philip/git/nebari-dev/nebari/nebari/destroy.py:179 in destroy_configuration                │
│                                                                                                  │
│   176 │                                                                                          │
│   177 │   # Populate stage_outputs to determine progress of deployment and                       │
│   178 │   # get credentials to kubernetes and keycloak context                                   │
│ ❱ 179 │   stage_outputs = gather_stage_outputs(config)                                           │
│   180 │                                                                                          │
│   181 │   with timer(logger, "destroying Nebari"):                                               │
│   182 │   │   status = destroy_stages(stage_outputs, config)                                     │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ config = ordereddict([('project_name', 'pmeier-aws-test'), ('provider', 'aws'), ('domain',   │ │
│ │          'nebari.pystiche.org'), ('certificate', ordereddict([('type', 'lets-encrypt'),      │ │
│ │          ('acme_email', 'pmeier@quansight.com'), ('acme_server',                             │ │
│ │          'https://acme-v02.api.letsencrypt.org/directory')])), ('security',                  │ │
│ │          ordereddict([('authentication', ordereddict([('type', 'password')])), ('keycloak',  │ │
│ │          ordereddict([('initial_root_password', 'Nv4T6wwrvXt9KmEF')]))])),                   │ │
│ │          ('default_images', ordereddict([('jupyterhub',                                      │ │
│ │          'quay.io/nebari/nebari-jupyterhub:2023.5.1'), ('jupyterlab',                        │ │
│ │          'quay.io/nebari/nebari-jupyterlab:2023.5.1'), ('dask_worker',                       │ │
│ │          'quay.io/nebari/nebari-dask-worker:2023.5.1')])), ('storage',                       │ │
│ │          ordereddict([('conda_store', '200Gi'), ('shared_filesystem', '200Gi')])), ('theme', │ │
│ │          ordereddict([('jupyterhub', ordereddict([('hub_title', 'Nebari - pmeier-aws-test'), │ │
│ │          ('hub_subtitle', 'Your open source data science platform, hosted on Amazon Web      │ │
│ │          Services'), ('welcome', 'Welcome! Learn about Nebari\'s features and configurations │ │
│ │          in <a href="https://www.nebari.dev/docs">the documentation</a>. If you have any     │ │
│ │          questions or feedback, reach the team on <a                                         │ │
│ │          href="https://www.nebari.dev/docs/community#getting-support">Nebari\'s support      │ │
│ │          forums</a>.'), ('logo',                                                             │ │
│ │          'https://raw.githubusercontent.com/nebari-dev/nebari-design/main/logo-mark/horizon… │ │
│ │          ('display_version', True), ('version', 'v2023.5.2.dev2+g8bd56f90')]))])),           │ │
│ │          ('helm_extensions', []), ('monitoring', ordereddict([('enabled', True)])),          │ │
│ │          ('argo_workflows', ordereddict([('enabled', True)])), ('kbatch',                    │ │
│ │          ordereddict([('enabled', True)])), ('cdsdashboards', ordereddict([('enabled',       │ │
│ │          True), ('cds_hide_user_named_servers', True), ('cds_hide_user_dashboard_servers',   │ │
│ │          False)])), ('terraform_state', ordereddict([('type', 'remote')])), ('namespace',    │ │
│ │          'dev'), ('nebari_version', '2023.5.2.dev2+g8bd56f90'), ('amazon_web_services',      │ │
│ │          ordereddict([('region', 'us-west-2'), ('kubernetes_version', '1.24'),               │ │
│ │          ('node_groups', ordereddict([('general', ordereddict([('instance', 'm5.2xlarge'),   │ │
│ │          ('min_nodes', 1), ('max_nodes', 1)])), ('user', ordereddict([('instance',           │ │
│ │          'm5.xlarge'), ('min_nodes', 1), ('max_nodes', 5), ('single_subnet', False)])),      │ │
│ │          ('worker', ordereddict([('instance', 'm5.xlarge'), ('min_nodes', 1), ('max_nodes',  │ │
│ │          5), ('single_subnet', False)]))]))])), ('profiles', ordereddict([('jupyterlab',     │ │
│ │          [ordereddict([('display_name', 'Small Instance'), ('description', 'Stable           │ │
│ │          environment with 2 cpu / 8 GB ram'), ('default', True), ('kubespawner_override',    │ │
│ │          ordereddict([('cpu_limit', 2), ('cpu_guarantee', 1.5), ('mem_limit', '8G'),         │ │
│ │          ('mem_guarantee', '5G')]))]), ordereddict([('display_name', 'Medium Instance'),     │ │
│ │          ('description', 'Stable environment with 4 cpu / 16 GB ram'),                       │ │
│ │          ('kubespawner_override', ordereddict([('cpu_limit', 4), ('cpu_guarantee', 3),       │ │
│ │          ('mem_limit', '16G'), ('mem_guarantee', '10G')]))])]), ('dask_worker',              │ │
│ │          ordereddict([('Small Worker', ordereddict([('worker_cores_limit', 2),               │ │
│ │          ('worker_cores', 1.5), ('worker_memory_limit', '8G'), ('worker_memory', '5G'),      │ │
│ │          ('worker_threads', 2)])), ('Medium Worker', ordereddict([('worker_cores_limit', 4), │ │
│ │          ('worker_cores', 3), ('worker_memory_limit', '16G'), ('worker_memory', '10G'),      │ │
│ │          ('worker_threads', 4)]))]))])), ('environments',                                    │ │
│ │          ordereddict([('environment-dask.yaml', ordereddict([('name', 'dask'), ('channels',  │ │
│ │          ['conda-forge']), ('dependencies', ['python=3.10.8', 'ipykernel=6.21.0',            │ │
│ │          'ipywidgets==7.7.1', 'nebari-dask ==2023.5.1', 'python-graphviz=0.20.1',            │ │
│ │          'pyarrow=10.0.1', 's3fs=2023.1.0', 'gcsfs=2023.1.0', 'numpy=1.23.5',                │ │
│ │          'numba=0.56.4', 'pandas=1.5.3', ordereddict([('pip', ['kbatch==0.4.1'])])])])),     │ │
│ │          ('environment-dashboard.yaml', ordereddict([('name', 'dashboard'), ('channels',     │ │
│ │          ['conda-forge']), ('dependencies', ['python=3.10',                                  │ │
│ │          'cdsdashboards-singleuser=0.6.3', 'cufflinks-py=0.17.3', 'dash=2.8.1',              │ │
│ │          'geopandas=0.12.2', 'geopy=2.3.0', 'geoviews=1.9.6', 'gunicorn=20.1.0',             │ │
│ │          'holoviews=1.15.4', 'ipykernel=6.21.2', 'ipywidgets=8.0.4', 'jupyter=1.0.0',        │ │
│ │          'jupyterlab=3.6.1', 'jupyter_bokeh=3.0.5', 'matplotlib=3.7.0',                      │ │
│ │          'nebari-dask==2023.5.1', 'nodejs=18.12.1', 'numpy', 'openpyxl=3.1.1',               │ │
│ │          'pandas=1.5.3', 'panel=0.14.3', 'param=1.12.3', 'plotly=5.13.0',                    │ │
│ │          'python-graphviz=0.20.1', 'rich=13.3.1', 'streamlit=1.9.0', 'sympy=1.11.1',         │ │
│ │          'voila=0.4.0', 'pip=23.0', ordereddict([('pip',                                     │ │
│ │          ['streamlit-image-comparison==0.0.3', 'noaa-coops==0.2.1',                          │ │
│ │          'dash_core_components==2.0.0', 'dash_html_components==2.0.0'])])])]))]))])          │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│ /home/philip/git/nebari-dev/nebari/nebari/destroy.py:69 in gather_stage_outputs                  │
│                                                                                                  │
│    66 │                                                                                          │
│    67 │   stage_outputs["stages/07-kubernetes-services"] = _terraform_init_output(               │
│    68 │   │   directory="stages/07-kubernetes-services",                                         │
│ ❱  69 │   │   input_vars=input_vars.stage_07_kubernetes_services(stage_outputs, config),         │
│    70 │   )                                                                                      │
│    71 │                                                                                          │
│    72 │   stage_outputs["stages/08-nebari-tf-extensions"] = _terraform_init_output(              │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ _terraform_init_output = functools.partial(<function deploy at 0x7f42464e6670>,              │ │
│ │                          terraform_init=True, terraform_import=True, terraform_apply=False,  │ │
│ │                          terraform_destroy=False)                                            │ │
│ │                 config = ordereddict([('project_name', 'pmeier-aws-test'), ('provider',      │ │
│ │                          'aws'), ('domain', 'nebari.pystiche.org'), ('certificate',          │ │
│ │                          ordereddict([('type', 'lets-encrypt'), ('acme_email',               │ │
│ │                          'pmeier@quansight.com'), ('acme_server',                            │ │
│ │                          'https://acme-v02.api.letsencrypt.org/directory')])), ('security',  │ │
│ │                          ordereddict([('authentication', ordereddict([('type',               │ │
│ │                          'password')])), ('keycloak', ordereddict([('initial_root_password', │ │
│ │                          'Nv4T6wwrvXt9KmEF')]))])), ('default_images',                       │ │
│ │                          ordereddict([('jupyterhub',                                         │ │
│ │                          'quay.io/nebari/nebari-jupyterhub:2023.5.1'), ('jupyterlab',        │ │
│ │                          'quay.io/nebari/nebari-jupyterlab:2023.5.1'), ('dask_worker',       │ │
│ │                          'quay.io/nebari/nebari-dask-worker:2023.5.1')])), ('storage',       │ │
│ │                          ordereddict([('conda_store', '200Gi'), ('shared_filesystem',        │ │
│ │                          '200Gi')])), ('theme', ordereddict([('jupyterhub',                  │ │
│ │                          ordereddict([('hub_title', 'Nebari - pmeier-aws-test'),             │ │
│ │                          ('hub_subtitle', 'Your open source data science platform, hosted on │ │
│ │                          Amazon Web Services'), ('welcome', 'Welcome! Learn about Nebari\'s  │ │
│ │                          features and configurations in <a                                   │ │
│ │                          href="https://www.nebari.dev/docs">the documentation</a>. If you    │ │
│ │                          have any questions or feedback, reach the team on <a                │ │
│ │                          href="https://www.nebari.dev/docs/community#getting-support">Nebar… │ │
│ │                          support forums</a>.'), ('logo',                                     │ │
│ │                          'https://raw.githubusercontent.com/nebari-dev/nebari-design/main/l… │ │
│ │                          ('display_version', True), ('version',                              │ │
│ │                          'v2023.5.2.dev2+g8bd56f90')]))])), ('helm_extensions', []),         │ │
│ │                          ('monitoring', ordereddict([('enabled', True)])),                   │ │
│ │                          ('argo_workflows', ordereddict([('enabled', True)])), ('kbatch',    │ │
│ │                          ordereddict([('enabled', True)])), ('cdsdashboards',                │ │
│ │                          ordereddict([('enabled', True), ('cds_hide_user_named_servers',     │ │
│ │                          True), ('cds_hide_user_dashboard_servers', False)])),               │ │
│ │                          ('terraform_state', ordereddict([('type', 'remote')])),             │ │
│ │                          ('namespace', 'dev'), ('nebari_version',                            │ │
│ │                          '2023.5.2.dev2+g8bd56f90'), ('amazon_web_services',                 │ │
│ │                          ordereddict([('region', 'us-west-2'), ('kubernetes_version',        │ │
│ │                          '1.24'), ('node_groups', ordereddict([('general',                   │ │
│ │                          ordereddict([('instance', 'm5.2xlarge'), ('min_nodes', 1),          │ │
│ │                          ('max_nodes', 1)])), ('user', ordereddict([('instance',             │ │
│ │                          'm5.xlarge'), ('min_nodes', 1), ('max_nodes', 5), ('single_subnet', │ │
│ │                          False)])), ('worker', ordereddict([('instance', 'm5.xlarge'),       │ │
│ │                          ('min_nodes', 1), ('max_nodes', 5), ('single_subnet',               │ │
│ │                          False)]))]))])), ('profiles', ordereddict([('jupyterlab',           │ │
│ │                          [ordereddict([('display_name', 'Small Instance'), ('description',   │ │
│ │                          'Stable environment with 2 cpu / 8 GB ram'), ('default', True),     │ │
│ │                          ('kubespawner_override', ordereddict([('cpu_limit', 2),             │ │
│ │                          ('cpu_guarantee', 1.5), ('mem_limit', '8G'), ('mem_guarantee',      │ │
│ │                          '5G')]))]), ordereddict([('display_name', 'Medium Instance'),       │ │
│ │                          ('description', 'Stable environment with 4 cpu / 16 GB ram'),       │ │
│ │                          ('kubespawner_override', ordereddict([('cpu_limit', 4),             │ │
│ │                          ('cpu_guarantee', 3), ('mem_limit', '16G'), ('mem_guarantee',       │ │
│ │                          '10G')]))])]), ('dask_worker', ordereddict([('Small Worker',        │ │
│ │                          ordereddict([('worker_cores_limit', 2), ('worker_cores', 1.5),      │ │
│ │                          ('worker_memory_limit', '8G'), ('worker_memory', '5G'),             │ │
│ │                          ('worker_threads', 2)])), ('Medium Worker',                         │ │
│ │                          ordereddict([('worker_cores_limit', 4), ('worker_cores', 3),        │ │
│ │                          ('worker_memory_limit', '16G'), ('worker_memory', '10G'),           │ │
│ │                          ('worker_threads', 4)]))]))])), ('environments',                    │ │
│ │                          ordereddict([('environment-dask.yaml', ordereddict([('name',        │ │
│ │                          'dask'), ('channels', ['conda-forge']), ('dependencies',            │ │
│ │                          ['python=3.10.8', 'ipykernel=6.21.0', 'ipywidgets==7.7.1',          │ │
│ │                          'nebari-dask ==2023.5.1', 'python-graphviz=0.20.1',                 │ │
│ │                          'pyarrow=10.0.1', 's3fs=2023.1.0', 'gcsfs=2023.1.0',                │ │
│ │                          'numpy=1.23.5', 'numba=0.56.4', 'pandas=1.5.3',                     │ │
│ │                          ordereddict([('pip', ['kbatch==0.4.1'])])])])),                     │ │
│ │                          ('environment-dashboard.yaml', ordereddict([('name', 'dashboard'),  │ │
│ │                          ('channels', ['conda-forge']), ('dependencies', ['python=3.10',     │ │
│ │                          'cdsdashboards-singleuser=0.6.3', 'cufflinks-py=0.17.3',            │ │
│ │                          'dash=2.8.1', 'geopandas=0.12.2', 'geopy=2.3.0', 'geoviews=1.9.6',  │ │
│ │                          'gunicorn=20.1.0', 'holoviews=1.15.4', 'ipykernel=6.21.2',          │ │
│ │                          'ipywidgets=8.0.4', 'jupyter=1.0.0', 'jupyterlab=3.6.1',            │ │
│ │                          'jupyter_bokeh=3.0.5', 'matplotlib=3.7.0', 'nebari-dask==2023.5.1', │ │
│ │                          'nodejs=18.12.1', 'numpy', 'openpyxl=3.1.1', 'pandas=1.5.3',        │ │
│ │                          'panel=0.14.3', 'param=1.12.3', 'plotly=5.13.0',                    │ │
│ │                          'python-graphviz=0.20.1', 'rich=13.3.1', 'streamlit=1.9.0',         │ │
│ │                          'sympy=1.11.1', 'voila=0.4.0', 'pip=23.0', ordereddict([('pip',     │ │
│ │                          ['streamlit-image-comparison==0.0.3', 'noaa-coops==0.2.1',          │ │
│ │                          'dash_core_components==2.0.0',                                      │ │
│ │                          'dash_html_components==2.0.0'])])])]))]))])                         │ │
│ │          stage_outputs = {                                                                   │ │
│ │                          │   'stages/01-terraform-state': {},                                │ │
│ │                          │   'stages/02-infrastructure': {                                   │ │
│ │                          │   │   'kubeconfig_filename': {                                    │ │
│ │                          │   │   │   'sensitive': False,                                     │ │
│ │                          │   │   │   'type': 'string',                                       │ │
│ │                          │   │   │   'value': '/tmp/NEBARI_KUBECONFIG'                       │ │
│ │                          │   │   },                                                          │ │
│ │                          │   │   'kubernetes_credentials': {                                 │ │
│ │                          │   │   │   'sensitive': True,                                      │ │
│ │                          │   │   │   'type': [                                               │ │
│ │                          │   │   │   │   'object',                                           │ │
│ │                          │   │   │   │   {                                                   │ │
│ │                          │   │   │   │   │   'cluster_ca_certificate': 'string',             │ │
│ │                          │   │   │   │   │   'host': 'string',                               │ │
│ │                          │   │   │   │   │   'token': 'string'                               │ │
│ │                          │   │   │   │   }                                                   │ │
│ │                          │   │   │   ],                                                      │ │
│ │                          │   │   │   'value': {                                              │ │
│ │                          │   │   │   │   'cluster_ca_certificate': '-----BEGIN               │ │
│ │                          CERTIFICATE-----\nMIIC/jCCAeagAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMw… │ │
│ │                          │   │   │   │   'host':                                             │ │
│ │                          'https://B8FA2602F2698FBCCEBB2337B2858C59.gr7.us-west-2.eks.amazon… │ │
│ │                          │   │   │   │   'token':                                            │ │
│ │                          'k8s-aws-v1.aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8_QWN0aW9uPUdldENhbG… │ │
│ │                          │   │   │   }                                                       │ │
│ │                          │   │   },                                                          │ │
│ │                          │   │   'nfs_endpoint': {                                           │ │
│ │                          │   │   │   'sensitive': False,                                     │ │
│ │                          │   │   │   'type': 'string',                                       │ │
│ │                          │   │   │   'value':                                                │ │
│ │                          'fs-0812c9a2bc197f161.efs.us-west-2.amazonaws.com'                  │ │
│ │                          │   │   }                                                           │ │
│ │                          │   },                                                              │ │
│ │                          │   'stages/03-kubernetes-initialize': {},                          │ │
│ │                          │   'stages/04-kubernetes-ingress': {                               │ │
│ │                          │   │   'load_balancer_address': {                                  │ │
│ │                          │   │   │   'sensitive': False,                                     │ │
│ │                          │   │   │   'type': [                                               │ │
│ │                          │   │   │   │   'object',                                           │ │
│ │                          │   │   │   │   {'hostname': 'string', 'ip': 'string'}              │ │
│ │                          │   │   │   ],                                                      │ │
│ │                          │   │   │   'value': {                                              │ │
│ │                          │   │   │   │   'hostname':                                         │ │
│ │                          'a28be0f3c24674273abdfc88ab238ad5-1339728640.us-west-2.elb.amazona… │ │
│ │                          │   │   │   │   'ip': ''                                            │ │
│ │                          │   │   │   }                                                       │ │
│ │                          │   │   }                                                           │ │
│ │                          │   },                                                              │ │
│ │                          │   'stages/05-kubernetes-keycloak': {},                            │ │
│ │                          │   'stages/06-kubernetes-keycloak-configuration': {}               │ │
│ │                          }                                                                   │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│ /home/philip/git/nebari-dev/nebari/nebari/stages/input_vars.py:282 in                            │
│ stage_07_kubernetes_services                                                                     │
│                                                                                                  │
│   279 │   │   "name": config["project_name"],                                                    │
│   280 │   │   "environment": config["namespace"],                                                │
│   281 │   │   "endpoint": config["domain"],                                                      │
│ ❱ 282 │   │   "realm_id": stage_outputs["stages/06-kubernetes-keycloak-configuration"][          │
│   283 │   │   │   "realm_id"                                                                     │
│   284 │   │   ]["value"],                                                                        │
│   285 │   │   "node_groups": _calculate_node_groups(config),                                     │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │           config = ordereddict([('project_name', 'pmeier-aws-test'), ('provider', 'aws'),    │ │
│ │                    ('domain', 'nebari.pystiche.org'), ('certificate', ordereddict([('type',  │ │
│ │                    'lets-encrypt'), ('acme_email', 'pmeier@quansight.com'), ('acme_server',  │ │
│ │                    'https://acme-v02.api.letsencrypt.org/directory')])), ('security',        │ │
│ │                    ordereddict([('authentication', ordereddict([('type', 'password')])),     │ │
│ │                    ('keycloak', ordereddict([('initial_root_password',                       │ │
│ │                    'Nv4T6wwrvXt9KmEF')]))])), ('default_images', ordereddict([('jupyterhub', │ │
│ │                    'quay.io/nebari/nebari-jupyterhub:2023.5.1'), ('jupyterlab',              │ │
│ │                    'quay.io/nebari/nebari-jupyterlab:2023.5.1'), ('dask_worker',             │ │
│ │                    'quay.io/nebari/nebari-dask-worker:2023.5.1')])), ('storage',             │ │
│ │                    ordereddict([('conda_store', '200Gi'), ('shared_filesystem', '200Gi')])), │ │
│ │                    ('theme', ordereddict([('jupyterhub', ordereddict([('hub_title', 'Nebari  │ │
│ │                    - pmeier-aws-test'), ('hub_subtitle', 'Your open source data science      │ │
│ │                    platform, hosted on Amazon Web Services'), ('welcome', 'Welcome! Learn    │ │
│ │                    about Nebari\'s features and configurations in <a                         │ │
│ │                    href="https://www.nebari.dev/docs">the documentation</a>. If you have any │ │
│ │                    questions or feedback, reach the team on <a                               │ │
│ │                    href="https://www.nebari.dev/docs/community#getting-support">Nebari\'s    │ │
│ │                    support forums</a>.'), ('logo',                                           │ │
│ │                    'https://raw.githubusercontent.com/nebari-dev/nebari-design/main/logo-ma… │ │
│ │                    ('display_version', True), ('version', 'v2023.5.2.dev2+g8bd56f90')]))])), │ │
│ │                    ('helm_extensions', []), ('monitoring', ordereddict([('enabled',          │ │
│ │                    True)])), ('argo_workflows', ordereddict([('enabled', True)])),           │ │
│ │                    ('kbatch', ordereddict([('enabled', True)])), ('cdsdashboards',           │ │
│ │                    ordereddict([('enabled', True), ('cds_hide_user_named_servers', True),    │ │
│ │                    ('cds_hide_user_dashboard_servers', False)])), ('terraform_state',        │ │
│ │                    ordereddict([('type', 'remote')])), ('namespace', 'dev'),                 │ │
│ │                    ('nebari_version', '2023.5.2.dev2+g8bd56f90'), ('amazon_web_services',    │ │
│ │                    ordereddict([('region', 'us-west-2'), ('kubernetes_version', '1.24'),     │ │
│ │                    ('node_groups', ordereddict([('general', ordereddict([('instance',        │ │
│ │                    'm5.2xlarge'), ('min_nodes', 1), ('max_nodes', 1)])), ('user',            │ │
│ │                    ordereddict([('instance', 'm5.xlarge'), ('min_nodes', 1), ('max_nodes',   │ │
│ │                    5), ('single_subnet', False)])), ('worker', ordereddict([('instance',     │ │
│ │                    'm5.xlarge'), ('min_nodes', 1), ('max_nodes', 5), ('single_subnet',       │ │
│ │                    False)]))]))])), ('profiles', ordereddict([('jupyterlab',                 │ │
│ │                    [ordereddict([('display_name', 'Small Instance'), ('description', 'Stable │ │
│ │                    environment with 2 cpu / 8 GB ram'), ('default', True),                   │ │
│ │                    ('kubespawner_override', ordereddict([('cpu_limit', 2), ('cpu_guarantee', │ │
│ │                    1.5), ('mem_limit', '8G'), ('mem_guarantee', '5G')]))]),                  │ │
│ │                    ordereddict([('display_name', 'Medium Instance'), ('description', 'Stable │ │
│ │                    environment with 4 cpu / 16 GB ram'), ('kubespawner_override',            │ │
│ │                    ordereddict([('cpu_limit', 4), ('cpu_guarantee', 3), ('mem_limit',        │ │
│ │                    '16G'), ('mem_guarantee', '10G')]))])]), ('dask_worker',                  │ │
│ │                    ordereddict([('Small Worker', ordereddict([('worker_cores_limit', 2),     │ │
│ │                    ('worker_cores', 1.5), ('worker_memory_limit', '8G'), ('worker_memory',   │ │
│ │                    '5G'), ('worker_threads', 2)])), ('Medium Worker',                        │ │
│ │                    ordereddict([('worker_cores_limit', 4), ('worker_cores', 3),              │ │
│ │                    ('worker_memory_limit', '16G'), ('worker_memory', '10G'),                 │ │
│ │                    ('worker_threads', 4)]))]))])), ('environments',                          │ │
│ │                    ordereddict([('environment-dask.yaml', ordereddict([('name', 'dask'),     │ │
│ │                    ('channels', ['conda-forge']), ('dependencies', ['python=3.10.8',         │ │
│ │                    'ipykernel=6.21.0', 'ipywidgets==7.7.1', 'nebari-dask ==2023.5.1',        │ │
│ │                    'python-graphviz=0.20.1', 'pyarrow=10.0.1', 's3fs=2023.1.0',              │ │
│ │                    'gcsfs=2023.1.0', 'numpy=1.23.5', 'numba=0.56.4', 'pandas=1.5.3',         │ │
│ │                    ordereddict([('pip', ['kbatch==0.4.1'])])])])),                           │ │
│ │                    ('environment-dashboard.yaml', ordereddict([('name', 'dashboard'),        │ │
│ │                    ('channels', ['conda-forge']), ('dependencies', ['python=3.10',           │ │
│ │                    'cdsdashboards-singleuser=0.6.3', 'cufflinks-py=0.17.3', 'dash=2.8.1',    │ │
│ │                    'geopandas=0.12.2', 'geopy=2.3.0', 'geoviews=1.9.6', 'gunicorn=20.1.0',   │ │
│ │                    'holoviews=1.15.4', 'ipykernel=6.21.2', 'ipywidgets=8.0.4',               │ │
│ │                    'jupyter=1.0.0', 'jupyterlab=3.6.1', 'jupyter_bokeh=3.0.5',               │ │
│ │                    'matplotlib=3.7.0', 'nebari-dask==2023.5.1', 'nodejs=18.12.1', 'numpy',   │ │
│ │                    'openpyxl=3.1.1', 'pandas=1.5.3', 'panel=0.14.3', 'param=1.12.3',         │ │
│ │                    'plotly=5.13.0', 'python-graphviz=0.20.1', 'rich=13.3.1',                 │ │
│ │                    'streamlit=1.9.0', 'sympy=1.11.1', 'voila=0.4.0', 'pip=23.0',             │ │
│ │                    ordereddict([('pip', ['streamlit-image-comparison==0.0.3',                │ │
│ │                    'noaa-coops==0.2.1', 'dash_core_components==2.0.0',                       │ │
│ │                    'dash_html_components==2.0.0'])])])]))]))])                               │ │
│ │ final_logout_uri = 'https://nebari.pystiche.org/hub/login'                                   │ │
│ │ jupyterhub_theme = ordereddict([('hub_title', 'Nebari - pmeier-aws-test'), ('hub_subtitle',  │ │
│ │                    'Your open source data science platform, hosted on Amazon Web Services'), │ │
│ │                    ('welcome', 'Welcome! Learn about Nebari\'s features and configurations   │ │
│ │                    in <a href="https://www.nebari.dev/docs">the documentation</a>. If you    │ │
│ │                    have any questions or feedback, reach the team on <a                      │ │
│ │                    href="https://www.nebari.dev/docs/community#getting-support">Nebari\'s    │ │
│ │                    support forums</a>.'), ('logo',                                           │ │
│ │                    'https://raw.githubusercontent.com/nebari-dev/nebari-design/main/logo-ma… │ │
│ │                    ('display_version', True), ('version', 'v2023.5.2.dev2+g8bd56f90')])      │ │
│ │    stage_outputs = {                                                                         │ │
│ │                    │   'stages/01-terraform-state': {},                                      │ │
│ │                    │   'stages/02-infrastructure': {                                         │ │
│ │                    │   │   'kubeconfig_filename': {                                          │ │
│ │                    │   │   │   'sensitive': False,                                           │ │
│ │                    │   │   │   'type': 'string',                                             │ │
│ │                    │   │   │   'value': '/tmp/NEBARI_KUBECONFIG'                             │ │
│ │                    │   │   },                                                                │ │
│ │                    │   │   'kubernetes_credentials': {                                       │ │
│ │                    │   │   │   'sensitive': True,                                            │ │
│ │                    │   │   │   'type': [                                                     │ │
│ │                    │   │   │   │   'object',                                                 │ │
│ │                    │   │   │   │   {                                                         │ │
│ │                    │   │   │   │   │   'cluster_ca_certificate': 'string',                   │ │
│ │                    │   │   │   │   │   'host': 'string',                                     │ │
│ │                    │   │   │   │   │   'token': 'string'                                     │ │
│ │                    │   │   │   │   }                                                         │ │
│ │                    │   │   │   ],                                                            │ │
│ │                    │   │   │   'value': {                                                    │ │
│ │                    │   │   │   │   'cluster_ca_certificate': '-----BEGIN                     │ │
│ │                    CERTIFICATE-----\nMIIC/jCCAeagAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYD'+… │ │
│ │                    │   │   │   │   'host':                                                   │ │
│ │                    'https://B8FA2602F2698FBCCEBB2337B2858C59.gr7.us-west-2.eks.amazonaws.co… │ │
│ │                    │   │   │   │   'token':                                                  │ │
│ │                    'k8s-aws-v1.aHR0cHM6Ly9zdHMuYW1hem9uYXdzLmNvbS8_QWN0aW9uPUdldENhbGxlcklk… │ │
│ │                    │   │   │   }                                                             │ │
│ │                    │   │   },                                                                │ │
│ │                    │   │   'nfs_endpoint': {                                                 │ │
│ │                    │   │   │   'sensitive': False,                                           │ │
│ │                    │   │   │   'type': 'string',                                             │ │
│ │                    │   │   │   'value': 'fs-0812c9a2bc197f161.efs.us-west-2.amazonaws.com'   │ │
│ │                    │   │   }                                                                 │ │
│ │                    │   },                                                                    │ │
│ │                    │   'stages/03-kubernetes-initialize': {},                                │ │
│ │                    │   'stages/04-kubernetes-ingress': {                                     │ │
│ │                    │   │   'load_balancer_address': {                                        │ │
│ │                    │   │   │   'sensitive': False,                                           │ │
│ │                    │   │   │   'type': ['object', {'hostname': 'string', 'ip': 'string'}],   │ │
│ │                    │   │   │   'value': {                                                    │ │
│ │                    │   │   │   │   'hostname':                                               │ │
│ │                    'a28be0f3c24674273abdfc88ab238ad5-1339728640.us-west-2.elb.amazonaws.com… │ │
│ │                    │   │   │   │   'ip': ''                                                  │ │
│ │                    │   │   │   }                                                             │ │
│ │                    │   │   }                                                                 │ │
│ │                    │   },                                                                    │ │
│ │                    │   'stages/05-kubernetes-keycloak': {},                                  │ │
│ │                    │   'stages/06-kubernetes-keycloak-configuration': {}                     │ │
│ │                    }                                                                         │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'realm_id'


### Versions and dependencies used.

nebari: 2023.5.2.dev2+g8bd56f90

### Compute environment

AWS

### Integrations

_No response_

### Anything else?

_No response_
pavithraes commented 1 year ago

@viniciusdc - It may not be worth looking into this until the new extension mechanism is setup.

amardeepranu commented 1 year ago

Also getting this - I'm unable to destroy any stack on AWS, looks like its keycloak related - I'm also getting an Invalid Credentials error (from keycloak using the password grant option) on deploy unfortunately which is blocking me from deploying to AWS at all

│                                                                                                  │
│   279 │   │   "name": config["project_name"],                                                    │
│   280 │   │   "environment": config["namespace"],                                                │
│   281 │   │   "endpoint": config["domain"],                                                      │
│ ❱ 282 │   │   "realm_id": stage_outputs["stages/06-kubernetes-keycloak-configuration"][          │
│   283 │   │   │   "realm_id"                                                                     │
│   284 │   │   ]["value"],                                                                        │
│   285 │   │   "node_groups": _calculate_node_groups(config),                                     │
│