search

nebari-dev / nebari

🪴 Nebari - your open source data science platform
https://nebari.dev
BSD 3-Clause "New" or "Revised" License
279 stars 92 forks source link

[BUG] - Can't deploy Nebari locally on Ubuntu ARM64 #2162

Closed marcelovilla closed 7 months ago

marcelovilla commented 10 months ago

Describe the bug

I am using Parallels on a Mac with an Apple M1 chip to run an Ubuntu in an ARM64 VM. When trying to deploy Nebari locally on the VM for testing purposes, the deployment is always getting stuck at the kubernetes_keycloak stage. Here is the complete output:

[terraform]: Initializing the backend...
[terraform]: 
[terraform]: Successfully configured the backend "kubernetes"! Terraform will automatically
[terraform]: use this backend unless the backend configuration changes.
[terraform]: Upgrading modules...
[terraform]: - kubernetes-keycloak-helm in modules/kubernetes/keycloak-helm
[terraform]: 
[terraform]: Initializing provider plugins...
[terraform]: - Finding hashicorp/helm versions matching "2.1.2"...
[terraform]: - Finding hashicorp/kubernetes versions matching "2.20.0"...
[terraform]: - Finding latest version of hashicorp/random...
[terraform]: - Installing hashicorp/kubernetes v2.20.0...
[terraform]: - Installed hashicorp/kubernetes v2.20.0 (signed by HashiCorp)
[terraform]: - Installing hashicorp/random v3.6.0...
[terraform]: - Installed hashicorp/random v3.6.0 (signed by HashiCorp)
[terraform]: - Installing hashicorp/helm v2.1.2...
[terraform]: - Installed hashicorp/helm v2.1.2 (signed by HashiCorp)
[terraform]: 
[terraform]: Terraform has created a lock file .terraform.lock.hcl to record the provider
[terraform]: selections it made above. Include this file in your version control repository
[terraform]: so that Terraform can guarantee to make the same selections by default when
[terraform]: you run "terraform init" in the future.
[terraform]: 
[terraform]: Terraform has been successfully initialized!
[terraform]: 
[terraform]: You may now begin working with Terraform. Try running "terraform plan" to see
[terraform]: any changes that are required for your infrastructure. All Terraform commands
[terraform]: should now work.
[terraform]: 
[terraform]: If you ever set or change modules or backend configuration for Terraform,
[terraform]: rerun this command to reinitialize your working directory. If you forget, other
[terraform]: commands will detect it and remind you to do so if necessary.
[terraform]: 
[terraform]: Terraform used the selected providers to generate the following execution
[terraform]: plan. Resource actions are indicated with the following symbols:
[terraform]:   + create
[terraform]: 
[terraform]: Terraform will perform the following actions:
[terraform]: 
[terraform]:   # random_password.keycloak-nebari-bot-password will be created
[terraform]:   + resource "random_password" "keycloak-nebari-bot-password" {
[terraform]:       + bcrypt_hash = (sensitive value)
[terraform]:       + id          = (known after apply)
[terraform]:       + length      = 32
[terraform]:       + lower       = true
[terraform]:       + min_lower   = 0
[terraform]:       + min_numeric = 0
[terraform]:       + min_special = 0
[terraform]:       + min_upper   = 0
[terraform]:       + number      = true
[terraform]:       + numeric     = true
[terraform]:       + result      = (sensitive value)
[terraform]:       + special     = false
[terraform]:       + upper       = true
[terraform]:     }
[terraform]: 
[terraform]:   # module.kubernetes-keycloak-helm.helm_release.keycloak will be created
[terraform]:   + resource "helm_release" "keycloak" {
[terraform]:       + atomic                     = false
[terraform]:       + chart                      = "keycloak"
[terraform]:       + cleanup_on_fail            = false
[terraform]:       + create_namespace           = false
[terraform]:       + dependency_update          = false
[terraform]:       + disable_crd_hooks          = false
[terraform]:       + disable_openapi_validation = false
[terraform]:       + disable_webhooks           = false
[terraform]:       + force_update               = false
[terraform]:       + id                         = (known after apply)
[terraform]:       + lint                       = false
[terraform]:       + manifest                   = (known after apply)
[terraform]:       + max_history                = 0
[terraform]:       + metadata                   = (known after apply)
[terraform]:       + name                       = "keycloak"
[terraform]:       + namespace                  = "dev"
[terraform]:       + recreate_pods              = false
[terraform]:       + render_subchart_notes      = true
[terraform]:       + replace                    = false
[terraform]:       + repository                 = "https://codecentric.github.io/helm-charts"
[terraform]:       + reset_values               = false
[terraform]:       + reuse_values               = false
[terraform]:       + skip_crds                  = false
[terraform]:       + status                     = "deployed"
[terraform]:       + timeout                    = 300
[terraform]:       + values                     = [
[terraform]:           + <<-EOT
[terraform]:                 # https://github.com/codecentric/helm-charts/blob/keycloak-15.0.2/charts/keycloak/values.yaml
[terraform]:                 ingress:
[terraform]:                   # Helm chart (14.0 anyway) will only define Ingress records, not IngressRoute as required by Traefik, so
[terraform]:                   # we will need to define our own IngressRoute elsewhere.
[terraform]:                   enabled: false
[terraform]:                 
[terraform]:                 imagePullSecrets:
[terraform]:                   - name: "extcrcreds"
[terraform]:                 
[terraform]:                 extraEnv: |
[terraform]:                   - name: PROXY_ADDRESS_FORWARDING
[terraform]:                     value: "true"
[terraform]:                 
[terraform]:                 startupScripts:
[terraform]:                   keycloak.cli: |
[terraform]:                     {{- .Files.Get "scripts/keycloak.cli" | nindent 2 }}
[terraform]:                 
[terraform]:                   nebariadminuser.sh: |
[terraform]:                     /opt/jboss/keycloak/bin/add-user-keycloak.sh -r master -u root -p "{{ .Values.initial_root_password }}"
[terraform]:                     /opt/jboss/keycloak/bin/add-user-keycloak.sh -r master -u nebari-bot -p "{{ .Values.nebari_bot_password }}"
[terraform]:                 
[terraform]:                 extraInitContainers: |
[terraform]:                   - command:
[terraform]:                     - sh
[terraform]:                     - -c
[terraform]:                     - |
[terraform]:                       wget https://github.com/aerogear/keycloak-metrics-spi/releases/download/2.5.3/keycloak-metrics-spi-2.5.3.jar -P /data/ &&
[terraform]:                       export SHA256SUM=9b3f52f842a66dadf5ff3cc3a729b8e49042d32f84510a5d73d41a2e39f29a96 &&
[terraform]:                       if ! (echo "$SHA256SUM  /data/keycloak-metrics-spi-2.5.3.jar" | sha256sum -c)
[terraform]:                         then
[terraform]:                           echo "Error: Checksum not verified" && exit 1
[terraform]:                         else
[terraform]:                           chown 1000:1000 /data/keycloak-metrics-spi-2.5.3.jar &&
[terraform]:                           chmod 777 /data/keycloak-metrics-spi-2.5.3.jar
[terraform]:                       fi
[terraform]:                     image: busybox:1.31
[terraform]:                     name: initialize-spi-metrics-jar
[terraform]:                     securityContext:
[terraform]:                       runAsUser: 0
[terraform]:                     volumeMounts:
[terraform]:                       - name: metrics-plugin
[terraform]:                         mountPath: /data
[terraform]:                 
[terraform]:                 extraVolumeMounts: |
[terraform]:                   - name: metrics-plugin
[terraform]:                     mountPath: /opt/jboss/keycloak/providers/
[terraform]:                 
[terraform]:                 extraVolumes: |
[terraform]:                   - name: metrics-plugin
[terraform]:                     emptyDir: {}
[terraform]:             EOT,
[terraform]:           + jsonencode(
[terraform]:                 {
[terraform]:                   + nodeSelector = {
[terraform]:                       + "kubernetes.io/os" = "linux"
[terraform]:                     }
[terraform]:                   + postgresql   = {
[terraform]:                       + primary = {
[terraform]:                           + nodeSelector = {
[terraform]:                               + "kubernetes.io/os" = "linux"
[terraform]:                             }
[terraform]:                         }
[terraform]:                     }
[terraform]:                 }
[terraform]:             ),
[terraform]:           + jsonencode({}),
[terraform]:         ]
[terraform]:       + verify                     = false
[terraform]:       + version                    = "15.0.2"
[terraform]:       + wait                       = true
[terraform]:       + wait_for_jobs              = false
[terraform]: 
[terraform]:       + set {
[terraform]:           + name  = "initial_root_password"
[terraform]:           + value = "pteujv58rrurbgh8jmdqfnyj9b2ph0ma"
[terraform]:         }
[terraform]: 
[terraform]:       + set_sensitive {
[terraform]:           # At least one attribute in this block is (or was) sensitive,
[terraform]:           # so its contents will not be displayed.
[terraform]:         }
[terraform]:     }
[terraform]: 
[terraform]:   # module.kubernetes-keycloak-helm.kubernetes_manifest.keycloak-http will be created
[terraform]:   + resource "kubernetes_manifest" "keycloak-http" {
[terraform]:       + manifest = {
[terraform]:           + apiVersion = "traefik.containo.us/v1alpha1"
[terraform]:           + kind       = "IngressRoute"
[terraform]:           + metadata   = {
[terraform]:               + name      = "keycloak-http"
[terraform]:               + namespace = "dev"
[terraform]:             }
[terraform]:           + spec       = {
[terraform]:               + entryPoints = [
[terraform]:                   + "websecure",
[terraform]:                 ]
[terraform]:               + routes      = [
[terraform]:                   + {
[terraform]:                       + kind     = "Rule"
[terraform]:                       + match    = "Host(`172.18.1.100`) && PathPrefix(`/auth`) "
[terraform]:                       + services = [
[terraform]:                           + {
[terraform]:                               + name      = "keycloak-headless"
[terraform]:                               + namespace = "dev"
[terraform]:                               + port      = 80
[terraform]:                             },
[terraform]:                         ]
[terraform]:                     },
[terraform]:                 ]
[terraform]:             }
[terraform]:         }
[terraform]:       + object   = {
[terraform]:           + apiVersion = "traefik.containo.us/v1alpha1"
[terraform]:           + kind       = "IngressRoute"
[terraform]:           + metadata   = {
[terraform]:               + annotations                = (known after apply)
[terraform]:               + clusterName                = (known after apply)
[terraform]:               + creationTimestamp          = (known after apply)
[terraform]:               + deletionGracePeriodSeconds = (known after apply)
[terraform]:               + deletionTimestamp          = (known after apply)
[terraform]:               + finalizers                 = (known after apply)
[terraform]:               + generateName               = (known after apply)
[terraform]:               + generation                 = (known after apply)
[terraform]:               + labels                     = (known after apply)
[terraform]:               + managedFields              = (known after apply)
[terraform]:               + name                       = "keycloak-http"
[terraform]:               + namespace                  = "dev"
[terraform]:               + ownerReferences            = (known after apply)
[terraform]:               + resourceVersion            = (known after apply)
[terraform]:               + selfLink                   = (known after apply)
[terraform]:               + uid                        = (known after apply)
[terraform]:             }
[terraform]:           + spec       = {
[terraform]:               + entryPoints = [
[terraform]:                   + "websecure",
[terraform]:                 ]
[terraform]:               + routes      = [
[terraform]:                   + {
[terraform]:                       + kind        = "Rule"
[terraform]:                       + match       = "Host(`172.18.1.100`) && PathPrefix(`/auth`) "
[terraform]:                       + middlewares = (known after apply)
[terraform]:                       + priority    = (known after apply)
[terraform]:                       + services    = [
[terraform]:                           + {
[terraform]:                               + kind               = (known after apply)
[terraform]:                               + name               = "keycloak-headless"
[terraform]:                               + namespace          = "dev"
[terraform]:                               + passHostHeader     = (known after apply)
[terraform]:                               + port               = "80"
[terraform]:                               + responseForwarding = {
[terraform]:                                   + flushInterval = (known after apply)
[terraform]:                                 }
[terraform]:                               + scheme             = (known after apply)
[terraform]:                               + sticky             = {
[terraform]:                                   + cookie = {
[terraform]:                                       + httpOnly = (known after apply)
[terraform]:                                       + name     = (known after apply)
[terraform]:                                       + sameSite = (known after apply)
[terraform]:                                       + secure   = (known after apply)
[terraform]:                                     }
[terraform]:                                 }
[terraform]:                               + strategy           = (known after apply)
[terraform]:                               + weight             = (known after apply)
[terraform]:                             },
[terraform]:                         ]
[terraform]:                     },
[terraform]:                 ]
[terraform]:               + tls         = {
[terraform]:                   + certResolver = (known after apply)
[terraform]:                   + domains      = (known after apply)
[terraform]:                   + options      = {
[terraform]:                       + name      = (known after apply)
[terraform]:                       + namespace = (known after apply)
[terraform]:                     }
[terraform]:                   + secretName   = (known after apply)
[terraform]:                   + store        = {
[terraform]:                       + name      = (known after apply)
[terraform]:                       + namespace = (known after apply)
[terraform]:                     }
[terraform]:                 }
[terraform]:             }
[terraform]:         }
[terraform]:     }
[terraform]: 
[terraform]: Plan: 3 to add, 0 to change, 0 to destroy.
[terraform]: 
[terraform]: Changes to Outputs:
[terraform]:   + keycloak_credentials         = (sensitive value)
[terraform]:   + keycloak_nebari_bot_password = (sensitive value)
[terraform]: random_password.keycloak-nebari-bot-password: Creating...
[terraform]: random_password.keycloak-nebari-bot-password: Creation complete after 0s [id=none]
[terraform]: module.kubernetes-keycloak-helm.kubernetes_manifest.keycloak-http: Creating...
[terraform]: module.kubernetes-keycloak-helm.kubernetes_manifest.keycloak-http: Creation complete after 0s
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Creating...
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [10s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [20s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [30s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [40s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [50s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [1m0s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [1m10s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [1m20s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [1m30s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [1m40s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [1m50s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [2m0s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [2m10s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [2m20s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [2m30s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [2m40s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [2m50s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [3m0s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [3m10s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [3m20s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [3m30s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [3m40s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [3m50s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [4m0s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [4m10s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [4m20s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [4m30s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [4m40s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [4m50s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Still creating... [5m0s elapsed]
[terraform]: module.kubernetes-keycloak-helm.helm_release.keycloak: Creation complete after 5m3s [id=keycloak]
[terraform]: 
[terraform]: Apply complete! Resources: 3 added, 0 changed, 0 destroyed.
[terraform]: 
[terraform]: Outputs:
[terraform]: 
[terraform]: keycloak_credentials = <sensitive>
[terraform]: keycloak_nebari_bot_password = <sensitive>
/home/parallels/miniforge3/envs/nebari-dev/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.18.1.100'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
Attempt 1 failed connecting to keycloak master realm
/home/parallels/miniforge3/envs/nebari-dev/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.18.1.100'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
Attempt 2 failed connecting to keycloak master realm
/home/parallels/miniforge3/envs/nebari-dev/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.18.1.100'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
Attempt 3 failed connecting to keycloak master realm
/home/parallels/miniforge3/envs/nebari-dev/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.18.1.100'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
Attempt 4 failed connecting to keycloak master realm
/home/parallels/miniforge3/envs/nebari-dev/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.18.1.100'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
Attempt 5 failed connecting to keycloak master realm
/home/parallels/miniforge3/envs/nebari-dev/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.18.1.100'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
Attempt 6 failed connecting to keycloak master realm
/home/parallels/miniforge3/envs/nebari-dev/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.18.1.100'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
Attempt 7 failed connecting to keycloak master realm
/home/parallels/miniforge3/envs/nebari-dev/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.18.1.100'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
Attempt 8 failed connecting to keycloak master realm
/home/parallels/miniforge3/envs/nebari-dev/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.18.1.100'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
Attempt 9 failed connecting to keycloak master realm
/home/parallels/miniforge3/envs/nebari-dev/lib/python3.10/site-packages/urllib3/connectionpool.py:1100: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.18.1.100'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
Attempt 10 failed connecting to keycloak master realm
ERROR: unable to connect to keycloak master realm at url=https://172.18.1.100/auth/ with root credentials

When I inspect the pods, i get the following:

$ kubectl --kubeconfig /tmp/NEBARI_KUBECONFIG --namespace dev get pods
NAME                                      READY   STATUS             RESTARTS   AGE
keycloak-0                                0/1     Init:0/2           0          12m
keycloak-postgresql-0                     0/1     CrashLoopBackOff   7          12m
nebari-traefik-ingress-7b86b979b7-7glgr   1/1     Running            0          13m

And when I get the logs, I see the following:

$ kubectl --kubeconfig /tmp/NEBARI_KUBECONFIG --namespace dev logs keycloak-postgresql-0
standard_init_linux.go:228: exec user process caused: exec format error

Expected behavior

I would expect Nebari to deploy correctly.

OS and architecture in which you are running Nebari

Linux ubuntu-linux-22-04-02-desktop 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:29:11 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

How to Reproduce the problem?

On an ARM64 Linux system:

  1. Create the nebari-config.yaml file:

    nebari init local \
--project nebari-arm \
--domain example.com \
--auth-provider password \
--terraform-state=local

  2. Setup DNS record:

    sudo echo "172.18.1.100  example.com" | sudo tee -a /etc/hosts

  3. Deploy Nebari:

    nebari deploy -c nebari-config.yaml --disable-prompt

Command output

No response

Versions and dependencies used.

$ conda --version
conda 23.3.1

$ kubectl version
Client Version: v1.29.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.21.10
WARNING: version difference between client (1.29) and server (1.21) exceeds the supported minor version skew of +/-1

$ nebari --version
2023.11.1

Compute environment

kind

Integrations

No response

Anything else?

No response

krassowski commented 8 months ago

I do not think that ARM64 is relevant here because I am also running into this on Ubuntu running on x64. It was not the case when I tried deploying locally a few weeks back but now I am getting exact same error.

Adam-D-Lewis commented 7 months ago

The recent difficulty deploying locally is likely related to https://github.com/nebari-dev/nebari/issues/2311

Adam-D-Lewis commented 7 months ago

Closing since this seems to be resolved, feel free to reopen if still seeing it