Running qhub deploy -c qhub-config.yaml twice throws an error in local deployment[bug]

[Adam-D-Lewis]

I've deployed qhub as mentioned in our docs through running qhub deploy -c qhub-config.yaml and it completes successfully. After the initial deployment, I'd like to change qhub-config.yaml slightly and redeploy. However, when I run qhub deploy -c qhub-config.yaml the second time, I get the following error.

Deleting all files and directories beneath /home/balast/CodingProjects/qhub-deploys/2021-10-20-qhub-extension/infrastructure ...
Deleting all files and directories beneath /home/balast/CodingProjects/qhub-deploys/2021-10-20-qhub-extension/image ...
INFO:qhub.deploy:All qhub endpoints will be under https://github-actions.qhub.dev
INFO:qhub.provider.terraform:terraform init directory=infrastructure
INFO:qhub.provider.terraform: terraform at /tmp/terraform/1.0.0/terraform
[terraform]: Initializing modules...
[terraform]: - forwardauth in modules/kubernetes/forwardauth
[terraform]: - kubernetes-conda-store-mount in modules/kubernetes/nfs-mount
[terraform]: - kubernetes-conda-store-server in modules/kubernetes/services/conda-store
[terraform]: - kubernetes-ingress in modules/kubernetes/ingress
[terraform]: - kubernetes-initialization in modules/kubernetes/initialization
[terraform]: - kubernetes-nfs-mount in modules/kubernetes/nfs-mount
[terraform]: - kubernetes-nfs-server in modules/kubernetes/nfs-server
[terraform]: - my-prefect-server-deployment-extension in modules/kubernetes/services/extensions
[terraform]: - my-redis-deployment-extension in modules/kubernetes/services/extensions
[terraform]: - qhub in modules/kubernetes/services/meta/qhub
[terraform]: - qhub.external-container-reg in modules/kubernetes/services/extcr
[terraform]: - qhub.kubernetes-dask-gateway in modules/kubernetes/services/dask-gateway
[terraform]: - qhub.kubernetes-jupyterhub in modules/kubernetes/services/jupyterhub
[terraform]: - qhub.kubernetes-jupyterhub-ssh in modules/kubernetes/services/jupyterhub-ssh
[terraform]: 
[terraform]: Initializing the backend...
[terraform]: 
[terraform]: Initializing provider plugins...
[terraform]: - Finding hashicorp/helm versions matching "2.1.2"...
[terraform]: - Finding latest version of hashicorp/random...
[terraform]: - Finding latest version of hashicorp/tls...
[terraform]: - Finding hashicorp/kubernetes versions matching "2.3.2"...
[terraform]: - Finding hashicorp/kubernetes-alpha versions matching "0.3.2"...
[terraform]: - Installing hashicorp/random v3.1.0...
[terraform]: - Installed hashicorp/random v3.1.0 (signed by HashiCorp)
[terraform]: - Installing hashicorp/tls v3.1.0...
[terraform]: - Installed hashicorp/tls v3.1.0 (signed by HashiCorp)
[terraform]: - Installing hashicorp/kubernetes v2.3.2...
[terraform]: - Installed hashicorp/kubernetes v2.3.2 (signed by HashiCorp)
[terraform]: - Installing hashicorp/kubernetes-alpha v0.3.2...
[terraform]: - Installed hashicorp/kubernetes-alpha v0.3.2 (signed by HashiCorp)
[terraform]: - Installing hashicorp/helm v2.1.2...
[terraform]: - Installed hashicorp/helm v2.1.2 (signed by HashiCorp)
[terraform]: 
[terraform]: Terraform has created a lock file .terraform.lock.hcl to record the provider
[terraform]: selections it made above. Include this file in your version control repository
[terraform]: so that Terraform can guarantee to make the same selections by default when
[terraform]: you run "terraform init" in the future.
[terraform]: 
[terraform]: ╷
[terraform]: │ Warning: Additional provider information from registry
[terraform]: │ 
[terraform]: │ The remote registry returned warnings for
[terraform]: │ registry.terraform.io/hashicorp/kubernetes-alpha:
[terraform]: │ - Please do not rely on this provider for production use while we strive
[terraform]: │ towards project maturity.
[terraform]: │ https://github.com/hashicorp/terraform-provider-kubernetes-alpha#experimental-status
[terraform]: ╵
[terraform]: 
[terraform]: Terraform has been successfully initialized!
[terraform]: 
[terraform]: You may now begin working with Terraform. Try running "terraform plan" to see
[terraform]: any changes that are required for your infrastructure. All Terraform commands
[terraform]: should now work.
[terraform]: 
[terraform]: If you ever set or change modules or backend configuration for Terraform,
[terraform]: rerun this command to reinitialize your working directory. If you forget, other
[terraform]: commands will detect it and remind you to do so if necessary.
INFO:qhub.provider.terraform:terraform init took 6.432 [s]
INFO:qhub.provider.terraform:terraform apply directory=infrastructure targets=['module.kubernetes', 'module.kubernetes-initialization']
INFO:qhub.provider.terraform: terraform at /tmp/terraform/1.0.0/terraform
[terraform]: 
[terraform]: Terraform used the selected providers to generate the following execution
[terraform]: plan. Resource actions are indicated with the following symbols:
[terraform]:   + create
[terraform]: 
[terraform]: Terraform will perform the following actions:
[terraform]: 
[terraform]:   # module.kubernetes-initialization.kubernetes_namespace.main will be created
[terraform]:   + resource "kubernetes_namespace" "main" {
[terraform]:       + id = (known after apply)
[terraform]: 
[terraform]:       + metadata {
[terraform]:           + generation       = (known after apply)
[terraform]:           + name             = "dev"
[terraform]:           + resource_version = (known after apply)
[terraform]:           + uid              = (known after apply)
[terraform]:         }
[terraform]:     }
[terraform]: 
[terraform]: Plan: 1 to add, 0 to change, 0 to destroy.
[terraform]: module.kubernetes-initialization.kubernetes_namespace.main: Creating...
[terraform]: ╷
[terraform]: │ Warning: Resource targeting is in effect
[terraform]: │ 
[terraform]: │ You are creating a plan with the -target option, which means that the
[terraform]: │ result of this plan may not represent all of the changes requested by the
[terraform]: │ current configuration.
[terraform]: │ 
[terraform]: │ The -target option is not for routine use, and is provided only for
[terraform]: │ exceptional situations such as recovering from errors or mistakes, or when
[terraform]: │ Terraform specifically suggests to use it as part of an error message.
[terraform]: ╵
[terraform]: ╷
[terraform]: │ Warning: Applied changes may be incomplete
[terraform]: │ 
[terraform]: │ The plan was created with the -target option in effect, so some changes
[terraform]: │ requested in the configuration may have been ignored and the output values
[terraform]: │ may not be fully updated. Run the following command to verify that no other
[terraform]: │ changes are pending:
[terraform]: │     terraform plan
[terraform]: │ 
[terraform]: │ Note that the -target option is not suitable for routine use, and is
[terraform]: │ provided only for exceptional situations such as recovering from errors or
[terraform]: │ mistakes, or when Terraform specifically suggests to use it as part of an
[terraform]: │ error message.
[terraform]: ╵
[terraform]: ╷
[terraform]: │ Error: namespaces "dev" already exists
[terraform]: │ 
[terraform]: │   with module.kubernetes-initialization.kubernetes_namespace.main,
[terraform]: │   on modules/kubernetes/initialization/main.tf line 1, in resource "kubernetes_namespace" "main":
[terraform]: │    1: resource "kubernetes_namespace" "main" {
[terraform]: │ 
[terraform]: ╵

Problem encountered: Terraform error

I'm not sure why qhub tries to delete everything. It may be because we seem to be terraform init again even though we shouldn't be? Not sure yet though.

Additionally, the error above happens when qhub deploy is run a second time even without changing qhub-config.yaml in between running qhub deploy commands.

[Adam-D-Lewis]

After digging in a little, it appears the render step always deletes old rendered qhub deployments. Running qhub deploy --config qhub-config.yaml --disable-prompt --disable-render gets around the issue. Maybe a note should be added to the testing docs, since I imagine this is a common testing workflow.

[viniciusdc]

After digging in a little, it appears the render step always deletes old rendered qhub deployments. Running qhub deploy --config qhub-config.yaml --disable-prompt --disable-render gets around the issue. Maybe a note should be added to the testing docs, since I imagine this is a common testing workflow.

I think that we might be deleting some of the terraform state files in the process of rendering... which might be causing terraform to think that the resource needs to be created again. I've encountered the same error as well.

[viniciusdc]

@Adam-D-Lewis have you encountered this issue again?

[Adam-D-Lewis]

@viniciusdc I have not seen this issue in at least a few months. :tada:

[iameskild]

We also recently switched from a local deployment on minikube to kind 👍