Insufficient with Create Custom Resource with RBAC Service Account

[DewyattStanfield]

I've been trying to run the pre-install report and always receive insufficient with creating a custom resource in the namespace with the RBAC service account. I see in the log that the process creates a role and a service account but then fails on the can-i create and can-i delete lines, which to me, shows the service account created cannot create or delete the custom resource, but why do I receive insufficient on only the creation and the deletion shows sufficient in the report? 2022-01-07 16:34:12,429 - pre_install_utils.py[line:60] - INFO: deploy_manifest_file rc 0 action apply filepath /sas/utils/viya4-ark/pre_install_report/library/utils/viya-role.yaml datab'role.rbac.authorization.k8s.io/viyasrole created\n' 2022-01-07 16:34:13,101 - pre_install_utils.py[line:60] - INFO: deploy_manifest_file rc 0 action apply filepath /sas/utils/viya4-ark/pre_install_report/library/utils/crservice_acct.yaml datab'serviceaccount/crreader created\n' 2022-01-07 16:34:13,702 - pre_install_utils.py[line:60] - INFO: deploy_manifest_file rc 0 action apply filepath /sas/utils/viya4-ark/pre_install_report/library/utils/viya-rolebinding.yaml datab'rolebinding.rbac.authorization.k8s.io/viyasrolebinding created\n' 2022-01-07 16:34:14,206 - pre_install_utils.py[line:60] - INFO: deploy_manifest_file rc 0 action apply filepath /sas/utils/viya4-ark/pre_install_report/library/utils/crviya.yaml datab'viya.company.com/testcr created\n' 2022-01-07 16:34:14,431 - pre_install_utils.py[line:118] - INFO: can -i False can-i create viyas.company.com --as=system:serviceaccount:viya4:crreader 2022-01-07 16:34:14,623 - pre_install_utils.py[line:118] - INFO: can -i False can-i delete viyas.company.com --as=system:serviceaccount:viya4:crreader 2022-01-07 16:34:14,853 - pre_install_utils.py[line:60] - INFO: deploy_manifest_file rc 0 action delete filepath /sas/utils/viya4-ark/pre_install_report/library/utils/crviya.yaml datab'viya.company.com "testcr" deleted\n' 2022-01-07 16:34:15,029 - pre_install_utils.py[line:60] - INFO: deploy_manifest_file rc 0 action delete filepath /sas/utils/viya4-ark/pre_install_report/library/utils/viya-role.yaml datab'role.rbac.authorization.k8s.io "viyasrole" deleted\n' 2022-01-07 16:34:15,199 - pre_install_utils.py[line:60] - INFO: deploy_manifest_file rc 0 action delete filepath /sas/utils/viya4-ark/pre_install_report/library/utils/viya-rolebinding.yaml datab'rolebinding.rbac.authorization.k8s.io "viyasrolebinding" deleted\n' 2022-01-07 16:34:15,401 - pre_install_utils.py[line:60] - INFO: deploy_manifest_file rc 0 action delete filepath /sas/utils/viya4-ark/pre_install_report/library/utils/crservice_acct.yaml datab'serviceaccount "crreader" deleted\n'

[lasiva]

@DewyattStanfield Please attach the complete log generated. Also, are you using a KUBECONFIG with administrator rights in the cluster? Thanks.

[DewyattStanfield]

viya_pre_install_log_2022-01-07T16_34_05.log

The user in the KUBECONFIG is admin everywhere.

[DewyattStanfield]

I went through the creation lines one by one to see what was returned using kubectl apply -f - what I received was a successful creation of three of the four items. The line executed and the error returned are kubectl apply -f /sas/utils/viya4-ark/pre_install_report/library/utils/crviya.yaml error: unable to recognize "/sas/utils/viya4-ark/pre_install_report/library/utils/crviya.yaml": no matches for kind "Viya" in version "company.com/v1"

[lasiva]

Thank you.. for the additional information.

[lasiva]

Please attach the log generated with the --debug option. Also, please attach the html report also. Thanks

[DewyattStanfield]

Files attached viya_pre_install_report_2022-01-10T13_00_09.html.txt viya_pre_install_log_2022-01-10T13_00_09.log

[DewyattStanfield]

While troubleshooting I applied the file crdviya.yaml which creates the CustomResourceDefinition viyas.company.com

kubectl get CustomResourceDefinition viyas.company.com NAME CREATED AT viyas.company.com 2022-01-10T18:25:37Z

Then, I described the resource and saw this, is it a problem? The full describe result is attached below

Message: spec.preserveUnknownFields: Invalid value: true: must be false Reason: Violations Status: True

I will continue with my troubleshooting efforts, and I appreciate how quickly you are responding. If there is anything else you need from me, I will get it as soon as possible.

CustomResourceDefinition.txt

[erharb]

@DewyattStanfield it appears your cluster may be running k8s version 1.22? If so then v1beta1 native k8s resources are no longer supported, only v1, and Viya 4 ARK does not yet support 1.22. Viya 4 itself is adding 1.22 support in a future release cadence as well.

[DewyattStanfield]

@erharb I saw that the v1beta1 in the files hello-application.yaml and crviya.yaml, but at this time it's just a warning when applying that and doesn't actually error. It sounds as if to deploy Viya4 I would need to downgrade my cluster - is that what you are saying? If so, what version do you recommend?

As for this error specifically, I went through and applied all the files manually with success, and I then tried to run

kubectl auth can-i create viyas.company.com --as=system:serviceaccount:viya4:crreader

The error returned is

Warning: the server doesn't have a resource type 'viyas' in group 'company.com' Error from server (Forbidden): {"Code":{"Code":"Forbidden","Status":403},"Message":"clusters.management.cattle.io \"c-gjk8h\" is forbidden: User \"system:serviceaccount:viya4:crreader\" cannot get resource \"clusters\" in API group \"management.cattle.io\" at the cluster scope","Cause":null,"FieldName":""} (post selfsubjectaccessreviews.authorization.k8s.io)

I suppose it doesn't matter at this point whether or not we fix the issue, as after downgrading to deploy Viya4 might resolve the specific issue.

Thanks again for your help on this, and I'll let you know here if downgrading the cluster resolved the issue.

[erharb]

For Viya 4 currently released software, kubernetes versions 1.19-1.21 is currently supported. It's up to you on which version though if it's all the same to you higher is usually better since it will be supported longer from the current point in time.

From https://go.documentation.sas.com/doc/en/sasadmincdc/v_022/itopssr/n1ika6zxghgsoqn1mq4bck9dx695.htm?fromDefault=

Kubernetes
    Microsoft Azure Kubernetes Service (AKS) 1.19.x - 1.21.x.
    Kubernetes 1.22 and later are not supported.

A future release of Viya 4 should add 1.22 support

[DewyattStanfield]

Are you sure you saw v1.22? What I see is

kubectl version Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:59:11Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.11", GitCommit:"27522a29febbcc4badac257763044d0d90c11abd", GitTreeState:"clean", BuildDate:"2021-09-15T19:16:25Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

[erharb]

I had assumed 1.22 due to not seeing v1beta1 resources in the log that was attached which is why I asked just in case, thanks for confirming it is not 1.22.

[erharb]

@DewyattStanfield do you get any errors from manually running kubectl apply on the CRD yaml file (pre_install_report/library/utils/crdviya.yaml)?

[DewyattStanfield]

@erharb no, I was able to run through those. I did receive a warning for those files that had v1beta1, but it was a warning. The only error I received was

Warning: the server doesn't have a resource type 'viyas' in group 'company.com'
Error from server (Forbidden): {"Code":{"Code":"Forbidden","Status":403},"Message":"clusters.management.cattle.io "c-gjk8h" is forbidden: User "system:serviceaccount:viya4:crreader" cannot get resource "clusters" in API group "management.cattle.io" at the cluster scope","Cause":null,"FieldName":""} (post selfsubjectaccessreviews.authorization.k8s.io)

When I tried the kubectl auth can-i. To me, it looks like the user I am using in my KUBECONFIG is being denied, so I will try a different ID. It's strange to me because the ID in the KUBECONFIG file is a global admin, but I cannot see anything else that would be wrong here.

[DewyattStanfield]

Sorry for the multiple posts - but there was one thing when describing the CRD after applying the crdviya.yaml file. It was

    Message:               spec.preserveUnknownFields: Invalid value: true: must be false
    Reason:                Violations
    Status:                True

Is this an issue?

[erharb]

Thanks for the clarification, what's interesting is there's nothing defining spec.preserveUnknownFields in this project, and crviya.yaml itself is pretty barebones, so I'm not exactly sure how or why that could be causing it.

Perhaps it is worth looking into your permissions and user as you have stated something strange regarding that user.

[DewyattStanfield]

SO, if I try to run kubectl auth can-i create viyas.company.com without specifying the ccreader account, it works fine. If I specify --as=system:serviceaccount:viya4:crreader or even --as=system:serviceaccount:viya4:default, the error is thrown. So my user in KUBECONFIG is fine, it's only when specifying the serviceaccount that there's a problem.

I'll continue to look into why, and hopefully resolve the issue.

[kevinlinglesas]

Considering the age of this issue, unless this issue is still occurring with the latest supported versions of k8s, we are closing this issue. We have not had any further reported issues like this.