Cannot create new cloud credentials for custom node drivers

[jakefhyde]

Setup

• Rancher version: v2.7-head (7913e28)
• Browser type & version: Google Chrome: Version 111.0.5563.64 (Official Build) unknown (64-bit)

Describe the bug

When enabling some custom node drivers, the Cloud Credential creation page doesn't not show the correct default fields, and has an uneditable foo key.

To Reproduce

• Enable a custom node driver (in my case, cloud.ca)
• Try to create a cloud credential (the correct key (apiKey) should show up the first time).
• Delete the cloud credential
• Attempt to create another cloud credential

Result

An unremovable foo key is present

Expected Result

Correct field apiKey shows in UI

Screenshots

Additional context

I've noticed this on other cloud providers as well. As an aside, what populated these fields by default? Is it hard coded, or is it interpolated from available fields (from a list of token, apiKey, password, etc.), or is it coming from the backend somewhere and I've missed it?

[aalves08]

@richard-cox this topic is quite similar to our last discussion around extensions. I can reproduce this by enabling cloud.ca node driver and creating a new RKE2 cloud.ca cluster... It's supposed to "use" the resourceFields coming from cloudcacredentialconfig but I suspect that this won't work for RKE2 right?

also, this scenario happens because the KeyValue is getting null from resourceFields field on the schema.

[richard-cox]

There's a chance this has never worked. Node drivers that were implemented with RKE1 in mind appear enabled for RKE2 as well. Unless this worked in previous versions I don't think this is a 2.7.2 thing.

~An improvement would be to determine if we have a schema for the machine config associated with the provider (rke-machine-config.cattle.io.<provider name>config) before showing the provider icon when creating an rke2 cluster. We should be careful though with how we get the provider name, it should be determined in the same when way we create the machine configs. @nwmac does that approach sound sensible? For instance enabling the open stack node driver meant the schema rke-machine-config.cattle.io.openstackconfigs became available~

Edit: I'm talking junk, enabled node drivers have configs

@aalves08 You're talking about the cloudcacredentialConfig field from the cloudcredential resource? Are we not setting it correctly when created?

[bashofmann]

It should still be possible to add and use custom node drivers, also for RKE2.

[jakefhyde]

@richard-cox FWIW Openstack is technically built-in. There are probably 3 separate categories of node drivers we need to support, embedded, custom (in-tree, i.e. everything available by default third party node driver), and then custom out-of-tree drivers that anyone can write. I don't think it's too much of a stretch on the backend side to generate the custom in-tree schemas (add here: https://github.com/rancher/rancher/blob/01454eb896ea8e46f95daf096afa56c6ad0e3e41/pkg/data/management/machinedriver_data.go#L38-L56) and do best effort for out-of-tree. @snasovich what do you think?

[richard-cox]

@jakefhyde I did some digging, and Vince wrote some docs on this before he left - https://rancher.github.io/dashboard/code-base-works/machine-drivers#driver-binary which help me understand a tad.

• There's no custom credentials component for cloudca... so the generic credentials component is used
• That will try to find find the schema for the providers credentials config - v3/schemas/cloudcacredentialconfig
  • If it finds this it uses values from schema's resourceFields to populate the key value form (with uneditable keys as a schema exists)
  • If it fails to find the credential schema it falls back on the machine config schema (rke-machine-config.cattle.io.cloudcaconfig) and guestimates some fields from it's resourceFields collection

I'm not sure why prior to some actions (creating a cloudca cred), the form only shows apiKey. Following that action the form shows no fields, as Alex pointed out, because the schemas/cloudcacredentialconfig resourceFields contains a null value.

Interestingly I'm assuming the BE should translate node driver annotations like publicCredentialFields, privateCredentialFields, etc into schema values. For cloudca their driver does NOT contain any similar node driver annotations

[richard-cox]

Ok, worked out some of the weirdness.

1. Enable the cloudca node driver on a fresh install
2. Do not refresh the browser
   • Enabling node driver creates lots of provider specific CRDs. We get new dynamicSchema resource.create notifications for resources, specifically one with an id of cloudcacredentialconfig. We do NOT get a resource.create for schema though, so the UI doesn't know it exists
3. Try to create a cloud ca.
   • We have no credentials config schema, so fall back on a majorly trimmed down list from the cloudcaconfig schema (just apiKey).
   • The cloud credentials screen even warns Rancher has no built-in support for this driver. We've taken a guess, but consult the driver's documentation for the fields required for authentication.
4. Refresh... the cloudcacredentialconfig is now available, but contains no resourceFields

So...

• Issue 1: UI receives no resource.create socket message for the cloudcacredentialconfig schema
  • Need to solve this BE side - See https://github.com/rancher/rancher/issues/40995
• Issue 2: There are no custom credential form fields set resulting in the cloudcacredentialconfig schema's resourceFields being null.
  • Need to solve this FE side. Given that RKE1 only shows the api key field, it's probably falling back on the no schema flow, so we need to do the same in shell/cloud-credential/generic.vue data method (resourceFields if ( normanSchema ) { --> if ( normanSchema ?.resourceFields) {

[gaktive]

Related backend has moved to 2.7.x Q2 and likely will move to Q3.

@richard-cox & @jakefhyde shall I move this to 2.7.x for now if we're blocked on this? If this is needed, then we'd need to talk with Team 2.

[richard-cox]

Related backend has moved to 2.7.x Q2 and likely will move to Q3.

@richard-cox & @jakefhyde shall I move this to 2.7.x for now if we're blocked on this? If this is needed, then we'd need to talk with Team 2.

We should keep this one to address issue two from my comment above. The other issue from that comment is covered by the other ticket rancher/rancher side

[snasovich]

@richard-cox @gaktive , is there capacity / enough time to ensure no regressions to fix "issue 2" for 2.7.2?

[richard-cox]

In summary

1) This issue only covers issue 2 from https://github.com/rancher/dashboard/issues/8523#issuecomment-1485581660 2) Issue 1 from https://github.com/rancher/dashboard/issues/8523#issuecomment-1485581660 means we have to refresh after enabling the cloud ca node provider 3) The credentials page should show the apiKeys field (as per the credentials page when creating an cloud ca credentials in embedded rke1 interface)

[gaktive]

@snasovich it's a regression so we can get this into 2.7.2.

[snasovich]

@gaktive , oh, I didn't realize it's a regression. Thank you for giving it priority then!

[mantis-toboggan-md]

/backport v2.7.2

[gaktive]

Added QA/None since QA will test this in 2.7.2.

[richard-cox]

I'm adding the comment here for visibility. The fix unblocks users from creating credentials for cloud ca... however it falls back on a generic handler. There's a follow on issue for the maintainers of the node driver to either

• add credential fields to the cloud ca node driver (see https://rancher.github.io/dashboard/code-base-works/machine-drivers#driver-binary)
• add a custom component for cloud ca cloud credentials

[markusewalker]

QA TEST PLAN

Scenarios	Scenario	Test Case
1	As a standard user, attempt to create a cloud credential for node driver Cloud.ca, delete that credential and add additional cloud credentials
2	As a standard user, attempt to create a cloud credential for node driver Open Telekom Cloud, delete that credential and add additional cloud credentials

[markusewalker]

Tested this against v2.7-head https://github.com/rancher/rancher/commit/1712bf9f7116e25711fd76b99d7ce5672c2d4a23. Please see results below:

ENVIRONMENT DETAILS

• Rancher install: Helm
• Rancher version: v2.7-head https://github.com/rancher/rancher/commit/1712bf9f7116e25711fd76b99d7ce5672c2d4a23
• Browser: Chrome

TEST RESULT	Scenario	Test Case	Result
1	As a standard user, attempt to create a cloud credential for node driver Cloud.ca, delete that credential and add additional cloud credentials	:x:
2	As a standard user, attempt to create a cloud credential for node driver Open Telekom Cloud, delete that credential and add additional cloud credentials	:white_check_mark:

VALIDATION STEPS

Scenario 1

1. As a standard user, attempted to provision a downstream RKE2 node driver using Cloud.ca.
2. When creating a cloud credential, it never creates and it loops me back as shown below:
3. The cloud credential claims it is Saved, but it does not appear in the Cloud Credentials list in Cluster Management:

   ---

   Scenario 2

4. As a standard user, attempted to provision a downstream RKE2 node driver using Open Telekom Cloud.
5. Successfully able to create a cloud credential and it appears in the Cloud Credentials list in Cluster Management:
6. Validated that I am able to delete the cloud credential and create a new one as well with no issues.

NOTES I saw the exact behavior (as expected) in the backport issue, so I used the same GIFS here to showcase behavior seen. Reopening this issue due to the failures seen.

[richard-cox]

When the Cloud CA Node driver is enabled - browser is NOT refreshed (issue 1 from https://github.com/rancher/dashboard/issues/8523#issuecomment-1485581660)

• we do not receive the cloudcacredentialconfig schema resource.create websocket message (see https://github.com/rancher/rancher/issues/40995)
• we use the fields from the node driver but drop most of them (see shell/cloud-credential/generic.vue)
• only the API Key field is shown

When the page is refreshed after enabling the node driver (issue 2 from https://github.com/rancher/dashboard/issues/8523#issuecomment-1485581660)

• the schema for cloudcacredentialconfig is returned in the http fetch for schemas
• the schema does NOT have a populated resourceFields (see https://github.com/rancher/dashboard/issues/8523#issuecomment-1490508448). so we fall back on fields from node driver (as per above)
• however unlike above as we have a schema we retain lots of fields from the node driver
• lots of fields shown

In terms of when a user creates a credentials when the form shows lots of values...

• We make the POST to /v3/cloudcredentials and get a 201 Created response
• The response does not contain an id, so we fai
• However.... there's no id for the credential, so it's not actually saved and returned for the user to select

General OTC works fine because it's schema ("otccredentialconfig") has resourceFields defined

Issue Wise....

• Issue 1 remains unsolved - https://github.com/rancher/dashboard/issues/8523#issuecomment-1485581660
• Issue 2 needs this PR. This means we show the same fields in dashboard as we do in ember
• Additional Issue (https://github.com/rancher/dashboard/issues/8523#issuecomment-1500418400)
  • Unable to create a CloudCA credential.
  • BE - POST to /v3/cloudcredentials succeeds, but does not return or persist resource
  • https://github.com/rancher/rancher/issues/41745

Summary

• This will be blocked until we merged https://github.com/rancher/dashboard/pull/9046 and BE solves https://github.com/rancher/rancher/issues/41745
• Given date BE won't get through this
• Recommend moving to 2.7Q3

[richard-cox]

Note - Cloud CA credentials can still be created by creating a Node Template, either from the Cluster Manager Node Template list or going down the RKE1 cluster create route and adding it there.

[gaktive]

Pushing to Q4 since Q3 timeline is tight and backend ticket not scheduled.

Internal reference: SURE-6517

[gaktive]

Backend work remains unscheduled, need to check what's going on there.

[gaktive]

Backend remains unscheduled and UI capacity for 2.10.0 needs to be considered.