search

rancher / dashboard

The Rancher UI
https://rancher.com
Apache License 2.0
460 stars 261 forks source link

[BUG] Chart Installs - Setting global.cattle.psp.enabled=true during install is not respected in the actual chart install #8612

Closed nickwsuse closed 1 year ago

nickwsuse commented 1 year ago

Rancher Server Setup

Information about the Cluster

User Information

Describe the bug When installing the monitoring chart v102.0.0+up40.1.2 and setting the global.cattle.psp.enabled=true, the install completes but PSPs are not actually enabled. When looking for policies in the cluster there are only pod disruption budgets, and if you check the chart's yaml you'll find that the global.cattle.psp.enabled value is set to false.

To Reproduce

  1. Create an HA on v2.7.2-rc7
  2. Create a downstream AWS EC2 Cluster - RKE1 v1.24.10 (3 workers, 1 control plane, 1 etcd)
  3. Begin the install of the monitoring chart (v102.0.0+up40.1.2)
  4. Before completing the install, edit the chart's yaml and set global.cattle.psp.enabled to true
  5. Complete the chart install
  6. Once the chart is done installing, check the cluster's policies
  7. Check the monitoring chart's yaml

Result PSPs are not enabled for the monitoring chart, even though the global.cattle.psp.enabled value is set to true

Expected Result PSPs are enabled if global.cattle.psp.enabled is set to true on a cluster with k8s v1.24.10 or older

Additional context This is happening on both my local and a downstream cluster

nickwsuse commented 1 year ago

Update: I originally wrote this in regards to the Monitoring chart, but I'm seeing the same thing happening with Alerting Drivers and ~Istio~, likely other charts as well.

Update 2: Electric Boogaloo - turns out it DID work for Istio, but did not work for OPA Gatekeeper

I discovered this while trying to test https://github.com/rancher/dashboard/issues/8062 so not sure if it's related in some way (seems unlikely, but figured I'd mention it just in case).

geethub97 commented 1 year ago

We tested this using the helm CLI and were unable to reproduce the issue. The chart works as expected.

prachidamle commented 1 year ago

The root cause of this issue is a UI/Dashboard issue being tracked here: https://github.com/rancher/dashboard/issues/8566

prachidamle commented 1 year ago

Seems the above UI ticket is resolved and moved to-test.

prachidamle commented 1 year ago

Please test this with a new rc to make sure Dashboard fix is included. @nickwsuse

jiaqiluo commented 1 year ago

can be validated in rancher v2.7.2-rc8

ronhorton commented 1 year ago

Fail checked in rancher v2.7.2-rc8

  1. installed monitoring with psp.enabled: false
  2. chart values.yaml = false
  3. update monitoring with psp.enabled: true
  4. chart values.yaml = false

saved value always == false; user change to true is not saved in the chart values.yaml 2023-04-02_07-11-02.png 2023-04-02_07-13-04.png2023-04-02_07-18-42.png 2023-04-02_07-25-43.png

stormqueen1990 commented 1 year ago

I can partly reproduce the issue in Rancher v2.7.2-rc8.

Cluster type: local Kubernetes version: v1.24.4-rancher1-1

Steps used:

  1. Create an RKE local cluster with RKE 1.4.3 and Kubernetes v1.24.4.
  2. Install cert-manager: 
    helm upgrade --install cert-manager \
    jetstack/cert-manager \
    --namespace cert-manager \
    --create-namespace \
    --version v1.11.0 \
    --set installCRDs=true
  3. Install Rancher v2.7.2-rc8: 
    helm upgrade --install rancher \
    rancher-latest/rancher \
    --namespace cattle-system \
    --set hostname="${HOSTNAME}" \
    --set bootstrapPassword=admin \
    --version="2.7.2-rc8" \
    --create-namespace
  4. Install Rancher Monitoring v102.0.0+up40.1.2 with global.cattle.psp.enabled=false, but check and uncheck the checkbox before proceeding.
  5. Check Values YAML for both rancher-monitoring-crd and rancher-monitoring; CRD shows global.cattle.psp.enabled=true whilst main chart shows global.cattle.psp.enabled=false.

I am unable, however, to reproduce the same issue from the command-line. Steps used:

  1. Copy the values.yaml from Rancher UI.
  2. Paste it in a file.
  3. Install the Rancher Monitoring CRD chart v102.0.0+up40.1.2 from the tarball using the copied content as the values.yaml file: 
    helm upgrade --install rancher-monitoring-crd \
    --namespace cattle-monitoring-system \
    https://github.com/rancher/charts/raw/dev-v2.7/assets/rancher-monitoring-crd/rancher-monitoring-crd-102.0.0%2Bup40.1.2.tgz \
    --values /tmp/monitoring-values.yaml
  4. Check in the Rancher UI that no PSPs were installed: Rancher UI screenshot showing a ClusterRole, ClusterRoleBinding, ServiceAccount, and ConfigMap resources as well as the current status for the application Rancher Monitoring CRD

I realize that my values YAML is not equal to the one generated by the Rancher UI, but I am not sure how the values YAML for the CRD chart is generated by the UI.

mantis-toboggan-md commented 1 year ago

@ronhorton I'm a little confused by your screenshots; the third one seems to be contradictory to

saved value always == false; user change to true is not saved in the chart values.yaml

Is there a difference between chart values.yaml and the yaml shown from update>edit YAML ?

ronhorton commented 1 year ago

@mantis-toboggan-md yes! chart values.yaml = false and yaml shown from update>edit yaml = true.

mantis-toboggan-md commented 1 year ago

Sorry @ronhorton what I mean is, what is chart values.yaml in this context? Like where in the UI are those other screenshots from?

cnotv commented 1 year ago

For clearance, this issue works on RKE2 on DO and has been fixed with https://github.com/rancher/dashboard/issues/8566

Support is needed to create a downstream AWS EC2 Cluster - RKE1 v1.24.10 as I have no credentials at all.

https://user-images.githubusercontent.com/5009481/229757928-f69b3946-a8da-453a-a550-ca3ef39e76ca.mp4

cnotv commented 1 year ago

@nickwsuse could you provide an environment with downstream AWS EC2 Cluster - RKE1 v1.24.10?

nflynt commented 1 year ago

Following the steps laid out by @stormqueen1990, I was able to narrow down the requirements to trigger a mismatch between the PSP setting in the Monitoring chart.

Following the above, the network request for rancher-monitoring correctly specifies no value for global.cattle.psp.enabled, which is expected. (It defaults to false).

However, the request for rancher-monitoring-crd now incorrectly specifies global.cattle.psp.enabled=true, as though the "uncheck" operation was not consistently applied by the UI:

002_check_then_uncheck

So far I've only seen this behavior on Monitoring. OPA Gatekeeper seems unaffected. I haven't yet tried it on other charts that have the PSP checkbox.

nflynt commented 1 year ago

Update: the above also applies to chart upgrades. If you have installed the Monitoring chart with PSP set to true, then it is impossible (with the UI alone) to fully update the chart to set PSP to false. The setting will correctly apply to rancher-monitoring but will be ignored and rancher-monitoring-crd will keep its PSPs.

prachidamle commented 1 year ago

@mantis-toboggan-md some more info in comments above that can help reproduce

cnotv commented 1 year ago

@nflynt I am not able to reproduce that in the video, what am I doing wrong?

https://user-images.githubusercontent.com/5009481/229906157-0acf0412-9ffc-4e0a-8bd2-b8da4f24f66b.mp4

cnotv commented 1 year ago

Update: the above also applies to chart upgrades. If you have installed the Monitoring chart with PSP set to true, then it is impossible (with the UI alone) to fully update the chart to set PSP to false. The setting will correctly apply to rancher-monitoring but will be ignored and rancher-monitoring-crd will keep its PSPs.

Are these not 2 separated apps? Why should the UI manage both? Is it meant to be handled like that?

nflynt commented 1 year ago

@cnotv Possibly nothing. The steps you're following in the video look correct. I was running my test on v2.7.2-rc8, are you running on the dev branch? Could this perhaps already be fixed?

cnotv commented 1 year ago

I have been able to reproduce the issue with the environment provided by @nickwsuse

https://user-images.githubusercontent.com/5009481/229908687-36140f82-a472-4d3f-924f-d6f76d23c168.mp4

cnotv commented 1 year ago

@cnotv Possibly nothing. The steps you're following in the video look correct. I was running my test on v2.7.2-rc8, are you running on the dev branch? Could this perhaps already be fixed?

@nflynt I am running what is on master 2fd348e37 for us, these are the URLs if you wanna give it a try:

Doing a comparison of master and v2.7.2-rc8 I see relevant changes to the charts, so it may be that the reason to have different outcomes.

EDIT: Sorry I forgot the link.

mantis-toboggan-md commented 1 year ago

I've tried and failed many times over to reproduce the issue with checking and unchecking the enable PSP option, following the same steps exactly as in @cnotv's video, using the same environment, as well as one provided by @nflynt, as well as my own clusters. It seems to be very elusive.

https://user-images.githubusercontent.com/42977925/230157919-beb9f91a-cf7f-4561-a71b-d41b1bd379dd.mov

Update: the above also applies to chart upgrades. If you have installed the Monitoring chart with PSP set to true, then it is impossible (with the UI alone) to fully update the chart to set PSP to false. The setting will correctly apply to rancher-monitoring but will be ignored and rancher-monitoring-crd will keep its PSPs.

This I can consistently reproduce and will put a PR up for. It seems the more pressing concern - I think if my fix for the above doesn't resolve the check/uncheck thing others have seen, we should file an issue for 2.7 q2 to look into that further. But we definitely want to fix the upgrade behavior for 2.7.2.

mantis-toboggan-md commented 1 year ago

/backport v2.7.2

nflynt commented 1 year ago

Quick testing on v2.7.2-rc9: I am no longer able to reproduce the issues that I found with the PSP checkbox. The value is now correctly applied to both charts (CRD included) during installations and upgrades. :+1:

richard-cox commented 1 year ago

@mantis-toboggan-md Is this one fixed via https://github.com/rancher/dashboard/pull/8636?

ronhorton commented 1 year ago

Pass verified in rancher v2.7-head commit ID 5334be3

  1. created a k8s 1.24 downstream cluster
  2. installed rancher-monitoring:102.0.1+up40.1.2
  3. on screen 2 of install, checked `'enable psps' box
  4. completed install of monitoring
  5. started kubeshell on downstream cluster
  6. checked for psps > they are present as expected 2023-05-31_17-59-56.png