Helm3 in a minikube enviroment times out

[docktermj]

The following timeout occurs:

 porter install --cred spike-helm3 

installing spike-helm3-mysql...
executing install action from spike-helm3-mysql (installation: spike-helm3-mysql)
Install MySQL
/usr/local/bin/helm3 helm3 install my-mysql stable/mysql --version 1.6.2 --replace --set mysqlDatabase=mydb --set mysqlUser=mysql-admin
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: *******
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: *******
Error: Kubernetes cluster unreachable: Get "https://192.168.49.2:8443/version?timeout=32s": dial tcp 192.168.49.2:8443: i/o timeout
Error: exit status 1
err: exit status 1
Error: mixin execution failed: exit status 1
Error: 2 errors occurred:
    * container exit code: 1, message: <nil>. fetching outputs failed: error copying outputs from container: Error: No such container:path: 1b7c38509f55fdb4e6c48269d66b363982c8dabe9a267e5a08278789e1c0fb99:/cnab/app/outputs
    * required output mysql-password is missing and has no default

using porter.yaml with these contents:

# -----------------------------------------------------------------------------
# Spike: minikube
# Based on https://github.com/MChorfa/porter-helm3/blob/master/example/porter.yaml
# -----------------------------------------------------------------------------

# -----------------------------------------------------------------------------
# Metadata
# https://porter.sh/author-bundles/#bundle-metadata
# -----------------------------------------------------------------------------

description: Spike to try out porter with MySql
name: spike-helm3-mysql
version: 0.1.0
registry: senzing

# -----------------------------------------------------------------------------
# Mixins
# https://porter.sh/author-bundles/#mixins
# https://porter.sh/mixins/
# -----------------------------------------------------------------------------

mixins:
- helm3:
    clientVersion: "v3.3.4"
    repositories:
      stable:
        url: "https://charts.helm.sh/stable"

# -----------------------------------------------------------------------------
# Credentials
# https://porter.sh/author-bundles/#credentials
# https://porter.sh/credentials/
# -----------------------------------------------------------------------------

credentials:
- default: ~/.kube/config
  description: File with embedded server and user certificates to connect to a kubernetes cluster. Helm should have already been init'd on this cluster.
  name: kubeconfig
  path: /root/.kube/config

# -----------------------------------------------------------------------------
# Parameters
# https://porter.sh/author-bundles/#parameters
# https://porter.sh/parameters/
# -----------------------------------------------------------------------------

parameters:
- name: database-name
  type: string
  default: mydb
- name: mysql-user
  type: string
  default: mysql-admin
- name: namespace
  type: string
  default: ''
- name: mysql-name
  type: string
  default: my-mysql

# -----------------------------------------------------------------------------
# Custom Actions
# https://porter.sh/author-bundles/#custom-actions
# -----------------------------------------------------------------------------

customActions:
  status:
    description: "Get the status of a helm3 release"
    modifies: false
    stateless: true

# -----------------------------------------------------------------------------
# Bundle Actions
# https://porter.sh/author-bundles/#bundle-actions
# -----------------------------------------------------------------------------

# -- install ------------------------------------------------------------------

install:
  - helm3:
      description: "Install MySQL"
      name: "{{ bundle.parameters.mysql-name }}"
      chart: stable/mysql
      version: 1.6.2
      namespace: "{{ bundle.parameters.namespace }}"
      replace: true
      set:
        mysqlDatabase: "{{ bundle.parameters.database-name}}"
        mysqlUser: "{{ bundle.parameters.mysql-user }}"
      outputs:
      - name: mysql-root-password
        secret: "{{ bundle.parameters.mysql-name }}"
        key: mysql-root-password
      - name: mysql-password
        secret: "{{ bundle.parameters.mysql-name }}"
        key: mysql-password

# -- status -------------------------------------------------------------------

status:
  - helm3:
      description: "MySQL Status"
      arguments:
        - status
        - "{{ bundle.parameters.mysql-name }}"
      flags:
        o: yaml

# -- upgrade ------------------------------------------------------------------

upgrade:
  - helm3:
      description: "Upgrade MySQL"
      name: "{{ bundle.parameters.mysql-name }}"
      namespace: "{{ bundle.parameters.namespace }}"
      chart: stable/mysql
      version: 1.6.2
      outputs:
      - name: mysql-root-password
        secret: "{{ bundle.parameters.mysql-name }}"
        key: mysql-root-password
      - name: mysql-password
        secret: "{{ bundle.parameters.mysql-name }}"
        key: mysql-password

# -- uninstall ----------------------------------------------------------------

uninstall:
  - helm3:
      description: "Uninstall MySQL"
      namespace: "{{ bundle.parameters.namespace }}"
      releases:
        - "{{ bundle.parameters.mysql-name }}"

# -----------------------------------------------------------------------------
# outputs
# https://porter.sh/author-bundles/#parameters
# https://porter.sh/wiring/#outputs
# -----------------------------------------------------------------------------

outputs:
  - name: mysql-password
    description: "The mysql database password"
    type: string
    applyTo:
      - install
      - upgrade
    sensitive: true

when running:

1. Start cluster. Example:

   minikube start \
     --cpus 4 \
     --disk-size=50g  \
     --embed-certs \
     --memory 8192

2. Build bundle. Example:

   porter build

3. Create credentials Example:

   porter credentials generate spike-helm3

   1. Set:
      1. file path = /home/[username]/.kube/config

4. Install bundle. Example:

   porter install --cred spike-helm3

As this may be a bug in the Helm3 mixin, I've also posted at https://github.com/MChorfa/porter-helm3/issues/23 Once it's determined where the issue lies, I'll close one of the GitHub issues.

[carolynvs]

@docktermj Can you take a look at the kubeconfig that you are passing in as the credential when you run the bundle? Helm will use the context specified with current-context in that file. Check that the current context is actually the most recent cluster that you have created with minikube. Most likely, you have multiple entries in that config file and the current one is not pointing to an active cluster which is why helm can't connect.

You can run kubectl config current-context to see the name of the current context (it should be minikube).

Another command to try is kubectl get nodes --context minikube to make sure that the minikube context in your config file works.

Minikube with Porter does work and this is not related to the helm3 mixin. Usually when problems arise it is either due to the ip address not being reachable from inside a container, or the kubeconfig isn't set up with the proper cluster.

[docktermj]

Hi @carolynvs !

I've watched your videos. Thanks for that information; very helpful.

I'm following the instructions at https://github.com/Senzing/kubernetes-demo/tree/913532392574425d73bd28fdd39fa6e4273e4243/porter/spike-minikube

When doing those instructions, I also ran the following:

$ kubectl config current-context
minikube

and

$ kubectl get nodes --context minikube
NAME       STATUS   ROLES                  AGE     VERSION
minikube   Ready    control-plane,master   8m24s   v1.21.2

I'm still seeing the error:

Install MySQL
/usr/local/bin/helm3 helm3 install my-mysql stable/mysql --version 1.6.2 --replace --set mysqlDatabase=mydb --set mysqlUser=mysql-admin
:
Error: Kubernetes cluster unreachable: Get "https://192.168.49.2:8443/version?timeout=32s": dial tcp 192.168.49.2:8443: i/o timeout

When I run the following from my workstation, it runs just fine:

$ helm install my-mysql stable/mysql --version 1.6.2 --replace --set mysqlDatabase=mydb --set mysqlUser=mysql-admin
NAME: my-mysql
LAST DEPLOYED: Thu Jul 29 09:56:13 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
MySQL can be accessed via port 3306 on the following DNS name from within your cluster:
my-mysql.default.svc.cluster.local

To get your root password run:

    MYSQL_ROOT_PASSWORD=$(kubectl get secret --namespace default my-mysql -o jsonpath="{.data.mysql-root-password}" | base64 --decode; echo)

To connect to your database:

1. Run an Ubuntu pod that you can use as a client:

    kubectl run -i --tty ubuntu --image=ubuntu:16.04 --restart=Never -- bash -il

2. Install the mysql client:

    $ apt-get update && apt-get install mysql-client -y

3. Connect using the mysql cli, then provide your password:
    $ mysql -h my-mysql -p

To connect to your database directly from outside the K8s cluster:
    MYSQL_HOST=127.0.0.1
    MYSQL_PORT=3306

    # Execute the following command to route the connection:
    kubectl port-forward svc/my-mysql 3306

    mysql -h ${MYSQL_HOST} -P${MYSQL_PORT} -u root -p${MYSQL_ROOT_PASSWORD}

The contents of my ~/.kube/config are:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tL...
    extensions:
    - extension:
        last-update: Thu, 29 Jul 2021 09:46:23 EDT
        provider: minikube.sigs.k8s.io
        version: v1.22.0
      name: cluster_info
    server: https://192.168.49.2:8443
  name: minikube
contexts:
- context:
    cluster: minikube
    extensions:
    - extension:
        last-update: Thu, 29 Jul 2021 09:46:23 EDT
        provider: minikube.sigs.k8s.io
        version: v1.22.0
      name: context_info
    namespace: default
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
  user:
    client-certificate-data: LS0tL...
    client-key-data: LS0tL...

In know it might be an imposition, but can you try the instructions at https://github.com/Senzing/kubernetes-demo/tree/913532392574425d73bd28fdd39fa6e4273e4243/porter/spike-minikube to see if the issue is re-creatable?

[docktermj]

@carolynvs I do need to mention that in the instructions at https://github.com/Senzing/kubernetes-demo/tree/913532392574425d73bd28fdd39fa6e4273e4243/porter/spike-minikube are currently in a branch https://github.com/Senzing/kubernetes-demo/tree/issue-69.dockter.1 So after cloning the repository you'll have to

git checkout issue-69.dockter.1

[carolynvs]

I'm glad that the videos were helpful! 💯

I'd be happy to walk through the instructions and see if it reproduces for me. Thanks for including that, it really helps.

[docktermj]

@carolynvs Wonderful!

Here's what I'm running with:

$ porter version
porter v0.38.4 (3d6d0f77)

$ minikube version
minikube version: v1.22.0
commit: a03fbcf166e6f74ef224d4a63be4277d017bb62e

They both should be pretty recent.

[carolynvs]

I followed the instructions and was able to install the bundle successfully. Let's step back and consider how the bundle is executing, and its connectivity to your minikube cluster.

Porter connects to either the local docker daemon or to a remote host specified with DOCKER_HOST and then runs the bundle inside a container. Since your cluster is local to your machine and will not have a public ip address, you should be using the local docker daemon. Are you using docker desktop for mac? Running docker directly on linux?

Minikube could be running in a few different modes (e.g. using virtual box, or hyperkit), where it creates a VM and then runs docker containers for the kubernetes components. The cluster API is exposed over the VM's ip address.

Seems like we are having connectivity trouble from inside that docker container where the bundle is running to the ip address of the minikube VM.

Can you run the following commands to test it out?

1. Get the ip address of your cluster

   minikube ip

2. Start an interactive shell session inside the porter bundle container:

   docker run --rm -it --entrypoint sh senzing/spike-helm3-mysql-installer:v0.1.0

3. Check if you can access the cluster api from inside the bundle's container:

   curl -k https://MINIKUBE_IP:8443

You should see an api response like the following if connectivity between the VM and the container is working:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {

  },
  "code": 403
}

[docktermj]

Hi @carolynvs!

I'm delighted to hear it worked for you! There's hope!

I'm running directly on a brand new linux laptop using Pop!_OS 20.04 LTS (similar to Ubuntu).

Following your instructions:

$ minikube ip
192.168.49.2

$ docker run --rm -it --entrypoint sh senzing/spike-helm3-mysql-installer:v0.1.0
# curl -k https://192.168.49.2:8443
curl: (7) Failed to connect to 192.168.49.2 port 8443: Connection timed out

I also ran

$ echo $DOCKER_HOST

which returned nothing (i.e. not set).

Any tips on my next step?

Thank you in advance.

[docktermj]

I tried this, but I'm guessing you already know it would fail:

$ export DOCKER_HOST=192.168.49.2

$ docker run --rm -it --entrypoint sh senzing/spike-helm3-mysql-installer:v0.1.0
docker: Cannot connect to the Docker daemon at tcp://192.168.49.2:2375. Is the docker daemon running?.
See 'docker run --help'.

[carolynvs]

Oh yeah sorry, DOCKER_HOST is one of a set of environment variables that tell the docker client to connect to a remote docker daemon. I mentioned it only to make sure that you weren't using it, since that would make connecting to minikube impossible.

So that connection timeout means that we've isolated the problem to being connectivity between your docker daemon and the minikube VM. Which vm provider are you using with minikube? Virtual Box? kvm?

[docktermj]

So it look like there's something different about the linux minikube environment. This was demonstrated by @carolynvs's suggestion to change the command shown above to the following:

docker run --rm -it  --net host --entrypoint sh senzing/spike-helm3-mysql-installer:v0.1.0

(i.e. add a --net host parameter)

Then running this command in the docker container is successful.

 curl -k https://192.168.49.2:8443

[carolynvs]

Okay I've reproduced the timeout! By default, minikube on linux will use the docker driver when available. Minikube creates a separate docker network named minikube and then runs kubernetes there. This network is not accessible from other containers that you run with docker (such as a porter bundle).

One workaround is to run the following command to have docker run containers inside the minikube docker environment.

eval $(minikube docker-env)

After running that command, when you start up another container with docker, it now has access to the minikube cluster api.

Since this is just for developers to try things out, it seems like an okay workaround. The side effect is that when they run docker ps they will see the containers associated with the cluster, instead of them being isolated.

Long term I can see porter allowing the docker network to be configurable, such that you could set a config or environment variable to tell porter to run bundles with the minikube docker network connected.

[docktermj]

Hi @carolynvs,

So are you saying to first run

minikube ssh

Then in the Minikube session run

eval $(minikube docker-env)

or are you suggesting something different (i.e. run the eval command on the host system)?

[carolynvs]

You should try running just the eval command on the host machine. It will set a couple docker environment variables so that porter knows to connect to the docker daemon hosted by minikube instead of the one on your host.

[docktermj]

Hi @carolynvs!

I incorporated eval $(minkube docker-env) into the instructions

I'm now seeing a different error:

$ eval $(minikube docker-env)

$ porter install --cred spike-helm3
installing spike-helm3-mysql...

Error: 2 errors occurred:
    * cannot inspect image senzing/spike-helm3-mysql-installer:v0.1.0: Error response from daemon: Client sent an HTTP request to an HTTPS server.
    * required output mysql-password is missing and has no default

For reference:

$ minikube docker-env
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.39.144:2376"
export DOCKER_CERT_PATH="/home/senzing/.minikube/certs"
export MINIKUBE_ACTIVE_DOCKERD="minikube"

# To point your shell to minikube's docker-daemon, run:
# eval $(minikube -p minikube docker-env)

[carolynvs]

@docktermj I wasn't having success dealing with the inconsistent access from the minikube cluster api from another docker container. Coming up with a workaround for minikube was taking longer than adding a feature, so I've submitted a PR that will add support for specifying that the bundle should connect to a specific docker network (such as host or minikube). This should account not only for how minikube does things differently based on OS and driver, but also accommodate other dev tools like k3d.

https://github.com/getporter/porter/pull/1698