search

nebari-dev / nebari

🪴 Nebari - your open source data science platform
https://nebari.dev
BSD 3-Clause "New" or "Revised" License
279 stars 89 forks source link

[BUG] - Local deploy on Arm based Mac fails #2300

Closed pt247 closed 2 months ago

pt247 commented 7 months ago

Describe the bug

I am deploying Nebari locally on the Mac ARM (M3). Deployment is failing with the error: ERROR: After stage=04-kubernetes-ingress unable to connect to ingress host=172.18.1.100 port=80

Expected behavior

Local deployment should finish.

OS and architecture in which you are running Nebari

macOS Sonoma Version 14.3.1 Apple M3

How to Reproduce the problem?

mkdir nebari-local && cd nebari-local
[ $(uname -m) = arm64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.22.0/kind-darwin-arm64
chmod +x ./kind
nebari init local --project=thisisatest --domain github-actions.nebari.dev --auth-provider=password
sudo echo "172.18.1.100  github-actions.nebari.dev" | sudo tee -a /etc/hosts
nebari deploy -c nebari-config.yaml --disable-prompt
$ cat nebari-config.yaml

provider: local
namespace: dev
nebari_version: 0.1.dev1287+g6ae41e8
project_name: thisisatest
domain: github-actions.nebari.dev
ci_cd:
  type: none
terraform_state:
  type: remote
security:
  keycloak:
    initial_root_password: 93q1pmpf0rxhoqaxfwp5wn1jl5fh9kgq
  authentication:
    type: password
theme:
  jupyterhub:
    hub_title: Nebari - thisisatest
    welcome: Welcome! Learn about Nebari's features and configurations in <a href="https://www.nebari.dev/docs/welcome">the
      documentation</a>. If you have any questions or feedback, reach the team on
      <a href="https://www.nebari.dev/docs/community#getting-support">Nebari's support
      forums</a>.
    hub_subtitle: Your open source data science platform, hosted

Command output

$ nebari deploy -c nebari-config.yaml --disable-prompt
...
...
[terraform]: load_balancer_address = {
[terraform]:   "hostname" = ""
[terraform]:   "ip" = "172.18.1.100"
[terraform]: }
Attempt 1 failed to connect to tcp tcp://172.18.1.100:80
Attempt 2 failed to connect to tcp tcp://172.18.1.100:80
...
...
Attempt 10 failed to connect to tcp tcp://172.18.1.100:80
ERROR: After stage=04-kubernetes-ingress unable to connect to ingress host=172.18.1.100 port=80

That IP is an internal docker network IP and the port is also not exposed:

$ docker ps                  
CONTAINER ID   IMAGE                   COMMAND                  CREATED          STATUS          PORTS                       NAMES
de87fe05cef6   kindest/node:v1.23.13   "/usr/local/bin/entr…"   11 minutes ago   Up 11 minutes   127.0.0.1:52278->6443/tcp   test-cluster-control-plane

### Versions and dependencies used.

$ conda --version
conda 24.1.2

$ nebari --version 0.1.dev1287+g6ae41e8

$ kubectl version Client Version: v1.29.1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.23.13 WARNING: version difference between client (1.29) and server (1.23) exceeds the supported minor version skew of +/-1



### Compute environment

kind

### Integrations

_No response_

### Anything else?

_No response_
pt247 commented 7 months ago

I can see that IP Assigned to the load balancer pod on my machine is not 172.18.1.100.

nkaretnikov commented 4 months ago

@pt247 This is solved in https://github.com/nebari-dev/nebari/issues/1405#issuecomment-2101307799, see the docs linked in the post. You need to update the hosts file and also install/enable docker-mac-net-connect.

However, note that I couldn't really use Nebari after deploy because it crashed almost immediately due to high resource usage (on a laptop): https://github.com/nebari-dev/nebari/issues/1405#issuecomment-2115996181

nkaretnikov commented 4 months ago

For reference, the successful connection log looks like this (yes, the hostname was empty despite setting the domain in the nebari config file):

[terraform]: load_balancer_address = {
[terraform]:   "hostname" = ""
[terraform]:   "ip" = "172.18.1.100"
[terraform]: }
Attempt 1 succeeded to connect to tcp://172.18.1.100:80
Attempt 1 succeeded to connect to tcp://172.18.1.100:8786
Attempt 1 succeeded to connect to tcp://172.18.1.100:8022
Attempt 1 succeeded to connect to tcp://172.18.1.100:8023
Attempt 1 succeeded to connect to tcp://172.18.1.100:9080
Attempt 1 succeeded to connect to tcp://172.18.1.100:443
After stage=04-kubernetes-ingress kubernetes ingress available on tcp ports={80, 8786, 8022, 8023, 9080, 443}
DNS configured domain=nebari.example.com matches ingress ips=172.18.1.100

My hosts file:

172.18.1.100 nebari.example.com

Domain in the Nebari config:

domain: nebari.example.com

docker-mac-net-connect (from new docs linked above):

brew install chipmk/tap/docker-mac-net-connect
sudo brew services start chipmk/tap/docker-mac-net-connect
marcelovilla commented 2 months ago

I'm closing this issue as this has been solved in https://github.com/nebari-dev/nebari/issues/1405#issuecomment-2101307799