Non-interactive minikube addons configure?

krezovic commented 4 years ago

I'm trying to fully script a minikube start command so I can share it between developers.

I want to use metallb addon introduced recently, which requires LoadBalancer Start and End IPs (I can construct it with output from minikube ip) to be configured via minikube addons configure metallb.

Can this be set non-interactively? By default, it always spawns the input, and you have to input the addresses yourselves.

tstromberg commented 4 years ago

I think this is a great idea.

While there is no technical reason an addon could not be configured via minikube config or flags, there is not yet an automated way to plumb configuration options to addons.

DanielFallon commented 4 years ago

@krezovic did you make any progress implementing this? I'm looking into doing something similar and fetched/test-built the minikube source last night to begin hacking on the code.

Initial research/thoughts

Some of my initial thoughts/take-aways after looking at the addons/config codepaths:

The existing config cmd doesn't have any concept of configuration keys/values, it's just a script to ask questions and a set of changes to be run. (create secrets, etc)
Even if a script could configure things via the minikube config command line, this would still have some awkward behavior (e.g. secrets or truly developer configurable options like IP addresses to be used)

A Naive Design

Looking at this problem and the plethora of minikube addons and possible usecases for configuration, choosing a one-size fits all approach that can't be implemented incrementally will probably lead to bike-shedding and failure.

Instead of requiring all addons to be able to be immediately be configured via the command line or config file, I propose the following high-level changes:

Extend the --interactive flag for configuring addons
Create convenience functions to help addons use default values from profile configuration
Update cmd/config.go for addons to use the new convenience functions.

After these changes the workflow would be as follows:

minikube start -p profile --
# update profile configuration to include required values
minikube configure x --interactive=false
minikube addons enable x

This isn't my favorite set of changes, but it seems like a small, non-detrimental set of changes that accomplishes the goal without hamstring future improvements.

@tstromberg do you have any thoughts about this implementation or ideas about appropriate placement for the configuration? I wouldn't want my pull request to utterly clobber profile/name/config.json without buy-in from core maintainers.

I could also see it being worthwhile to create addon-specific files within the profile, but I'd like to hear other's thoughts.

sharifelgamal commented 4 years ago

I have no issue adding information to config.json so long as backwards compatibility is preserved. If adding a new file is the easier answer, then by all means go for it.

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

r4j4h commented 3 years ago

I know this is old, but in response to "metallb" add-on specifically, I poked around this after finding my configs getting reset sometimes, and not being able to trick it by echoing newlines, I wanted to share what I learned about customizing the start/end IPs incase it helps someone.

I found they are stored as part of the KubernetesConfig type in minikube, so it's stored in your minikube profile directory (e.g. ~/.minikube/profiles/minikube/config.json if default or ~/.minikube/profiles/*/config.json). Here's an example using jq to illustrate how to pre-configure the addresses, after which running configure will report a green happy checkmark, instead of asking for the IP ranges:

$ minikube start
$ minikube addons enable metallb
$ cat ~/.minikube/profiles/minikube/config.json | jq '.KubernetesConfig.LoadBalancerStartIP="192.168.103.1"' | jq '.KubernetesConfig.LoadBalancerEndIP="192.168.103.100"' > ~/.minikube/profiles/minikube/config.json.tmp && mv ~/.minikube/profiles/minikube/config.json.tmp ~/.minikube/profiles/minikube/config.json 
$ minikube addons configure metallb
    ▪ Using image metallb/speaker:v0.8.2
    ▪ Using image metallb/controller:v0.8.2
✅  metallb was successfully configured

This approach will work for automating start-end IP ranges, which should support most minikube's users needs.

If one wants or needs to use CIDR format for addresses or alternative metallb configurations, they will be overwritten unless you customize the addon files and build your minikube. So if you need/want that, then for now I'd probably suggest disabling the metallb addon and installing metallb manually, otherwise you will need to overwrite it by-design.

cscotti commented 3 years ago

Same approach as r4j4h but with minikube ip range. In that way, there is no extra network route to set locally. In my case it permit to access to the dashboard with a local ip (and not through minikube proxy).

$ minikube start

$ METALLB_IP_PREFIX_RANGE=$(minikube ip | sed -r 's/(.*)./\1/')
$ cat ~/.minikube/profiles/minikube/config.json  \
| jq '.KubernetesConfig.LoadBalancerStartIP="'${METALLB_IP_PREFIX_RANGE}105'"' \
| jq '.KubernetesConfig.LoadBalancerEndIP="'${METALLB_IP_PREFIX_RANGE}120'"' \
> ~/.minikube/profiles/minikube/config.json.tmp && mv ~/.minikube/profiles/minikube/config.json.tmp ~/.minikube/profiles/minikube/config.json 

$ minikube addons enable metallb
$ minikube addons configure metallb

$ minikube addons enable metrics-server
$ minikube addons enable dashboard

# patch dashboard service
$ kubectl get svc kubernetes-dashboard -n kubernetes-dashboard -o json \
| jq '.spec.type="LoadBalancer"' \
| jq '.spec.ports[0].port=8080' \
| kubectl apply -f -

# open the dashboard directly
$ DASHBOARD_URL=http://$(kubectl get svc kubernetes-dashboard  -n kubernetes-dashboard --no-headers | awk '{print $4}'):8080
$ echo "$DASHBOARD_URL"
$ xdg-open $DASHBOARD_URL

tonejito commented 2 years ago

I went on another route and created a new revision of the MetalLB ConfigMap after the addon was enabled:

https://github.com/kubernetes/minikube/issues/10307#issuecomment-1024575716

Hope it's useful for anyone

boostasbjorn commented 2 years ago

Modifying ~/.minikube/profiles/minikube/config.json stopped working for me in a recent minikube version. Using this now:

$ MINIKUBE_IP=$(minikube ip)
$ expect << _EOF_
spawn minikube addons configure metallb
expect "Enter Load Balancer Start IP:" { send "${MINIKUBE_IP%.*}.105\\r" }
expect "Enter Load Balancer End IP:" { send "${MINIKUBE_IP%.*}.120\\r" }
expect eof
_EOF_

This simulates inputting the data interactively.

marcelovcpereira commented 2 years ago

I also have a similar use case. We automate the whole configuration of the local environment in our company.... and configuring some stuff on minikube is important to achieve this. We, currently, use registry-creds externally, but as it is supported as official addon, we would love to use it this way. Unfortunately, we can't because we can't automate the configuration of the addon for our users (automatically specifying AWS instead of Azure, for example).

Would be great to see a way to programmatically configure any addons

gbajson commented 1 year ago

I have a similar issue, but with a different add-on:

$ minikube addons configure registry-creds

Do you want to enable AWS Elastic Container Registry? [y/n]: n

Do you want to enable Google Container Registry? [y/n]: n

Do you want to enable Docker Registry? [y/n]: y
-- Enter docker registry server url:

cjlarose commented 1 year ago

FWIW it's possible to automate the configuration for the registry-creds by creating the secret resources yourself as documented in the project's README.md

In other words, minikube addons configure registry-creds just creates four different secrets, one for each supported registry. minikube addons enable registry-creds creates the controller pod in such a way that it requires the presence of all four secrets, even if you care about only of them. For example, you can create the secret for GCR with your real credentials.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: registry-creds-gcr
  namespace: kube-system
  labels:
    app: registry-creds
    kubernetes.io/minikube-addons: registry-creds
    cloud: gcr
data:
  application_default_credentials.json: $(openssl base64 <<< "$APPLICATION_DEFAULT_CREDENTIALS_JSON" | tr -d '\n')
  gcrurl: $(echo -n 'https://gcr.io' | openssl base64)
type: Opaque
EOF

And then create the remaining 3 secrets with bogus data

cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: Secret
metadata:
  name: registry-creds-acr
  namespace: kube-system
  labels:
    app: registry-creds
    cloud: acr
    kubernetes.io/minikube-addons: registry-creds
data:
  ACR_CLIENT_ID: Y2hhbmdlbWU=
  ACR_PASSWORD: Y2hhbmdlbWU=
  ACR_URL: Y2hhbmdlbWU=
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
  name: registry-creds-dpr
  namespace: kube-system
  labels:
    app: registry-creds
    cloud: dpr
    kubernetes.io/minikube-addons: registry-creds
data:
  DOCKER_PRIVATE_REGISTRY_PASSWORD: Y2hhbmdlbWU=
  DOCKER_PRIVATE_REGISTRY_SERVER: Y2hhbmdlbWU=
  DOCKER_PRIVATE_REGISTRY_USER: Y2hhbmdlbWU=
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
  name: registry-creds-ecr
  namespace: kube-system
  labels:
    app: registry-creds
    cloud: ecr
    kubernetes.io/minikube-addons: registry-creds
data:
  AWS_ACCESS_KEY_ID: Y2hhbmdlbWU=
  AWS_SECRET_ACCESS_KEY: Y2hhbmdlbWU=
  AWS_SESSION_TOKEN: ""
  aws-account: Y2hhbmdlbWU=
  aws-assume-role: Y2hhbmdlbWU=
  aws-region: Y2hhbmdlbWU=
type: Opaque
EOF

After creating the secret resources, you can minikube addons enable registry-creds and the controller pod should spin up successfully and do its job.

kubernetes / minikube

Non-interactive minikube addons configure? #8283

Initial research/thoughts

A Naive Design