Update libs and add PSACT support

[a-blender]

Issue: https://github.com/rancher/terraform-provider-rke/issues/399

Problem

Currently the TFP RKE does not support setting PSA configuration on creation of K8s 1.25+ clusters.

Solution

What has been added

• [x] 1. Update RKE to [1.4.10](https://github.com/rancher/rke/releases/tag/v1.4.10
• [x] 2. Add the PSA-related options to the kube_api service block defined in the RKE docs (added - I have some important notes on test cases!)
• [x] Update PSACT docs and add/update config examples

Testing

Engineering Testing

Manual Testing

Tested/verified

• [x] Method 1 - Pass restricted policy via the kube-api.pod_security_configuration field

  services:
  kube-api:
  pod_security_configuration: restricted

• [ ] Method 2 - Pass a custom admission configuration yaml via the kube-api.admission_configuration field

• [x] Method 3: Pass the path to an admission configuration file via kube-api.extra_args.admission-control-config-file and set bind_mounts to mount that file from the cluster node into the kube API server container

Notes

Woo, this series of tests was tricky.

I verified that deploying a pod on a cluster with restricted PSA failed and values are getting passed correctly. We have a bug where not all kube-apiserver containers restart on all cp nodes after changing the PSA. Early in testing, I saw restricted policy warnings come up for a deployment when I had already switched to privileged. Could've been due to this bug, I had 2 cp nodes and the API server could've not been restarted on one of them.

I designed admission_configuration field in the TFP RKE provider as a string, emulating how the provider deals with the AuditLog policy. This is different from the rancher2 provider because we are converting back and forth between a map[string]interface and a v1 AdmissionConfiguration object where plugins is an unknown runtime object (kind of a nightmare) instead of two interfaces. So, a user can pass a full string of the Admission Configuration yaml they'd like to have applied to the cluster at pod level and RKE will do that.

Per method 3, Docker has this lovely thing where per the docs,

The file or directory does not need to exist on the Docker host already. It is created on demand if it does not yet exist.

Yes, it creates it. ...as a directory.

So, in order to use kube API extra_args in TF to mount a custom admission configuration file to the API server you have to update the API server Dockerfile to touch an empty file so it exists already. Otherwise,

unable to read admission control configuration from \"<NODE PATH>/admission.yaml\" [read <CONTAINER PATH>/admission.yaml: is a directory]"

For TFP RKE, I just had to verify the provider is passing the extra arg admission-control-config-file correctly which it is.

Automated Testing

• Test types added/modified:
  • Unit
  • None
• If "None" - Reason:
• If "None" - GH Issue/PR: LINK TO GH ISSUE/PR TO ADD TESTS

Summary: TODO

QA Testing Considerations

Regressions Considerations

TODO

Existing / newly added automated tests that provide evidence there are no regressions:

• TODO

Terraform

Click to Expand Terraform

[a-blender]

After extensive debugging on Method 2 - custom Admission Configuration, I've fixed the decoder and parsing issue so RKE is able to parse each AdmissionConfiguration plugin individually. However, due to a v1 plugin Configuration being an unknown Runtime object, there appears to be no way for TF provider to input a a yaml formatted admission configuration into the cluster.yml because It must be an Unknown Runtime object. It will be successfully parsed but applied to the cluster as bytes instead of yaml which fails to apply the plugin.

Fixing this requires either

1. Add local types in the provider - need to expand out to kube_api resource, gross overkill.
2. Update v1.AdmissionConfigurationPlugin type in RKE - butterfly effect, touches too many components in Rancher.

Due to this roadblock, we are pulling support for a custom Admission Configuration (2 PSACT methods will still be supported for users who want to implement this feature) into a separate issue. We will also add the future work of supporting a custom plugin of any type.