stellirin commented 3 years ago

We have a bare metal k3s cluster and we would like to replicate its configuration in Rancher Desktop. One example of what we would like to replicate is the use of --disable=traefik so that we can install Traefik ourselves. Reasons:

install v2 on versions of k3s where v1 is still the default
additional listeners to export a PostgreSQL service on port 5432

Not every configuration option makes sense to be configurable, but in this specific case it would map well to a checkbox in the UI (same for the other components that can be disabled with this flag).

Alternatives:

a simple text box that we can write the k3s flags and their options directly, accepting the risk that typos could cause k3s startup to fail
similar to above but to influence the Traefik installation directly (helm options etc.)

Check boxes to disable components and a second advanced text box for k3s flags seems like the best options for flexibility, and can be implemented separately.

JockDaRock commented 3 years ago

I am totally for this as working with Istio on rancher-desktop is difficult if you can't disable traefik. would be also interesting if you can disable the embedded (klipper?) lb as well :)

kt81 commented 3 years ago

It's nice to have the ability to configure any options. (alt. 1) I'd like to give k3s the option of --service-node-port-range but other people may not.

stellirin commented 2 years ago

note for me:

951 is taged for v1.0.0
926 is a workaround and was released in v0.7.0

On MacOS we can now manually edit ~/Library/Application\ Support/rancher-desktop/lima/0/lima.yaml to add any k3s server flags such as:

env:
  K3S_EXEC: --disable=traefik

cvstebut commented 2 years ago

Does anybody know how to set the variable on Windows / WSL? Thanks!

jandubois commented 2 years ago

On MacOS we can now manually edit ~/Library/Application\ Support/rancher-desktop/lima/0/lima.yaml

@stellirin Starting with RD 0.7.0 you should put any such customization into ~/Library/Application\ Support/rancher-desktop/lima/_config/override.yaml instead. That file will not be written to by RD itself, so customizations are protected from any changes RD makes to lima.yaml.

@cvstebut It is possible to make the same change with the WSL2 distro, but I don't know off-hand the best way to configure it. @mook-as should be able to provide an answer next week.

mook-as commented 2 years ago

There is no supported way of doing this on Windows yet. As a workaround, you can add K3S_EXEC=… to /etc/rc.conf (see the section around line 108, Service Configuration Variables); however, that will be clobbered when we update the base distribution (roughly, on Rancher Desktop updates).

Filed #1142 to properly support this.

cvstebut commented 2 years ago

Thanks a lot! For my use case this works great. Permanence is nice, but now I know where to configure it when needed. I have just successfully tested it:

opened the WSL distro "rancher-desktop" using the windows terminal
added a line to /etc/rc.conf K3S_EXEC="--disable=traefik --disable=servicelb"
"Reset Kubernetes" in the Rancher Desktop UI

Works as desired :-)

zerus83 commented 2 years ago

On MacOS we can now manually edit ~/Library/Application\ Support/rancher-desktop/lima/0/lima.yaml

@stellirin Starting with RD 0.7.0 you should put any such customization into ~/Library/Application\ Support/rancher-desktop/lima/_config/override.yaml instead. That file will not be written to by RD itself, so customizations are protected from any changes RD makes to lima.yaml.

@cvstebut It is possible to make the same change with the WSL2 distro, but I don't know off-hand the best way to configure it. @mook-as should be able to provide an answer next week.

Hi i tried using the override yaml method but keep getting errors.

override.yaml: env: K3S_EXEC: "--disable=traefik -flannel-backend=none"

Error encountered:

/lib/rc/sh/openrc-run.sh: /etc/environment: line 7: -flannel-backend=none: not found

k3s: error loading /etc/init.d/k3s
ERROR: k3s failed to start
any idea how to resolve this? thanks!

cvstebut commented 2 years ago

@zerus83 Could be just a spelling mistake: Try "--flannel-backend..." instead of just "-flannel-backend ..."

zerus83 commented 2 years ago

@zerus83 Could be just a spelling mistake: Try "--flannel-backend..." instead of just "-flannel-backend ..."

apologies , forgot to elaborate- actually i try a few combinations like the following:

K3S_EXEC: --cluster-cidr=10.66.0.0/16 --disable=traefik K3S_EXEC: "--cluster-cidr=10.66.0.0/16 --disable=traefik" K3S_EXEC: "--cluster-cidr=10.66.0.0/16 -disable=traefik"

whenever i added more than 1 option in the arguments, the 2nd option will have the error.

if i were to switch the order to K3S_EXEC: --disable=traefik --cluster-cidr=10.66.0.0/16, i will get the below error:

lib/rc/sh/openrc-run.sh: /etc/environment: line 7: --cluster-cidr=10.66.0.0/16: not found

i am using mac M1 silicon.

jandubois commented 2 years ago

Error encountered:

/lib/rc/sh/openrc-run.sh: /etc/environment: line 7: -flannel-backend=none: not found

This is a bug in /etc/init.d/k3s: it executes source /etc/environment, which is invalid because the file is not using shell syntax.

micahmo commented 2 years ago

@kt81

I'd like to give k3s the option of --service-node-port-range but other people may not.

+1, I have this use case as well. Note that you can use K3S_EXEC for this purpose.

K3S_EXEC="--kube-apiserver-arg service-node-port-range=1-65535"

jandubois commented 2 years ago

The option to disable traefik is included in the 1.1.0 release. A more general mechanism is still outstanding.

marcellodesales commented 2 years ago

🔧 Config

The settings described for the overrides.yaml worked flawlessly!
- https://github.com/rancher-sandbox/rancher-desktop/issues/578#issuecomment-997454430
Here's how I enabled feature gate CronJobTimeZone!
- https://kubernetes.io/docs/reference/command-line-tools-reference/_print/#feature-gates
- https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/

# https://github.com/rancher-sandbox/rancher-desktop/issues/578#issuecomment-996557436
env:
  K3S_EXEC: --kube-apiserver-arg --feature-gates=CronJobTimeZone=true

🔊 K3S Logs

Just make sure nothing went wrong by looking at the logs

tail -f ~/Library/Logs/rancher-desktop/k3s.log

I confirmed by verifying that the feature gates was seen on the logs: --feature-gates=CronJobTimeZone=true

time="2022-08-12T19:15:57Z" level=info msg="Acquiring lock file /var/lib/rancher/k3s/data/.lock"
time="2022-08-12T19:15:57Z" level=info msg="Preparing data dir /var/lib/rancher/k3s/data/230cb7b95d48672d3d564dfde9c7b6230bccc82b2a3298f565dc3cca2a06b2eb"
time="2022-08-12T19:15:59Z" level=info msg="Found ip 192.168.86.249 from iface rd0"
time="2022-08-12T19:15:59Z" level=info msg="Starting k3s v1.24.3+k3s1 (990ba0e8)"
time="2022-08-12T19:15:59Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
time="2022-08-12T19:15:59Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
time="2022-08-12T19:15:59Z" level=info msg="Database tables and indexes are up to date"
time="2022-08-12T19:15:59Z" level=info msg="Kine available at unix://kine.sock"
time="2022-08-12T19:15:59Z" level=info msg="Reconciling bootstrap data between datastore and disk"
time="2022-08-12T19:15:59Z" level=info msg="Running kube-apiserver --advertise-address=192.168.86.249 --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --etcd-servers=unix://kine.sock --feature-gates=CronJobTimeZone=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
time="2022-08-12T19:15:59Z" level=info msg="Tunnel server egress proxy mode: agent"

🐛 Bug setting CronJob timezones

As the feature gate is still in Alpha and it got enabled properly, given the validations, it fails to set the timezone for CronJobs.

still need to setup the timezone for the kube-controller-manager... :(
- https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
- Names starting with Etc/* as described at https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#time-zones

The CronJob "timecard-cron-job-local-localhost-prdt-ppd-loc" is invalid: spec.timeZone: Invalid value: "Etc/GMT+8": unknown time zone Etc/GMT+8

Even the default fails

The CronJob "timecard-cron-job-local-localhost-prdt-ppd-loc" is invalid: spec.timeZone: Invalid value: "Etc/UTC": unknown time zone Etc/UTC

❓ What's the current support for this feature?

@gaktive

Is there any way to set a value? or is it just broken?
Is there any workaround?

arslanakhtar61 commented 1 year ago

Would be nice to have a disabled checkbox in the rancher-desktop UI for KlipperLB, the same as with Traefik

stahir80td commented 1 year ago

I am trying to pass in multiple arguments to kubeapi-server. I edited /etc/rc.conf, I am seeing the values being passed in to the logs but not as I want them.

Goal: I need to pass in two parameters, not one.

Option 1: K3S_EXEC="--kube-apiserver-arg oidc-client-id=value1 oidc-issuer-url=value2"

Outcome: Only adds first parameter (oidc-client-id) to kubeapi-server param

Option 2: K3S_EXEC="--kube-apiserver-arg oidc-client-id=value1" K3S_EXEC="--kube-apiserver-arg oidc-issuer-url=value2"

Outcome: Only adds 2nd parameter (oidc-issuer-url) to kubeapi-server params

Any suggestions on how to pass in multiple parameters to kubeapi-server using /etc/rc.conf Many Thanks,

AxelTahmid commented 10 months ago

🐛 Bug setting CronJob timezones

still facing the unknown time zone issue, running 1.28.4

rbofa commented 8 months ago

We are not able to replicate rancher desktop with metallb. Traefik disabled on latest rancher desktop ui, but still servicelb now letting Loadbalancer type service to execute on linux rancher desktop ui.

Tried on rancher desktop cluster information UI screen, by setting the annotation k3s argument as ["--disable", "traefik","--disable", "servicelb" ] restrted the desktop. Still exposed service type: LoadBalancer is not working/reaching. Can see svclb- pods for the same service.

kind: Service apiVersion: v1 metadata: name: foo-service-2 spec: type: LoadBalancer selector: app: http-echo ports: port: 5678

rancher-sandbox / rancher-desktop

Feature: Allow advanced configuration of k3s from the UI #578

951 is taged for v1.0.0

926 is a workaround and was released in v0.7.0

🔧 Config

🔊 K3S Logs

🐛 Bug setting CronJob timezones

❓ What's the current support for this feature?

🐛 Bug setting CronJob timezones