Unable to change 'etcd Snapshot Backup Target' to local from S3 even though 'Allow User Override' is enabled

[gaktive]

Internal reference: SURE-4376 Reported in 2.6.4

After you create a Cluster with a RKE template, you can't switch to local from S3 for 'etcd Snapshot Backup Target' even though the field is set to 'Allow User Override' in the RKE template. According to our doc, it should be configurable at any time.

Overriding Template Settings https://rancher.com/docs/rancher/v2.6/en/admin-settings/rke-templates/overrides/

When any parameter is set as Allow User Override on the RKE template, it means that end users have to fill out those fields during cluster creation and they can edit those settings afterward at any time.

Regarding this issue, there are 2 behaviors:

1) You can't switch to local from S3 and you see this error.

This cluster is currently Updating.Failed to validate cluster: etcd s3 backup backend endpoint can't be empty 

2) You can't switch to local from S3 and you don't see the error.

Impact: You need 2 RKE templates that use local and S3 for 'etcd Snapshot Backup Target'. Without this issue, you can have just 1 RKE template because the field can be configurable even after a cluster created.

Repro steps:

1) Create a S3 bucket

For behavior 1:

a) Create a RKE template with the following settings:

• etcd Snapshot Backup Target: Allow User Override is enabled
• S3 Bucket Name: Allow User Override is enabled
• S3 Region: Allow User Override is enabled
• S3 Region Endpoint: Allow User Override is enabled
• S3 Access Key: Allow User Override is enabled
• S3 Secret Key: Allow User Override is enabled

b) Create a cluster with the RKE template c) After the cluster becomes Active, change 'etcd Snapshot Backup Target' to local and save the change. d) Confirm that you could not update the setting and you see the error

As for behavior 2:

a) Create a RKE template with the following settings:

• etcd Snapshot Backup Target: Allow User Override is enabled
• S3 Bucket Name: Allow User Override is disabled
• S3 Region: Allow User Override is disabled
• S3 Region Endpoint: Allow User Override is disabled
• S3 Access Key: Allow User Override is disabled
• S3 Secret Key: Allow User Override is disabled

b) Create a cluster with the RKE template c) After the cluster becomes Active, change 'etcd Snapshot Backup Target' to local and save the change. d) Confirm that you could not update the setting and no errors happened

Workaround: You create 2 RKE templates

Actual behavior: You can't change 'etcd Snapshot Backup Target' to local from S3 even though the field is set to 'Allow User Override'.

Expected behavior: You can change fields that are set to 'Allow User Override' at any time as our doc mentions.

[gaktive]

Upon discussion with @nwmac, we need to look at the UX and ensure that:

• if the override is set for S3, then we properly take that since it seems the logic is that if this is the only override, it's ignored.
• we need a banner to indicate how the overrides work.

[gaktive]

To discuss with @kwwii

[gaktive]

May have to transfer to backend. Will sync with @Jono-SUSE-Rancher

[gaktive]

Backend ticket recently filed: https://github.com/rancher/rancher/issues/46204 that's not scheduled so it's not just UI.