search

sassoftware / viya4-ark

SAS Viya Administration Resource Kit for Containerized Deployments. Tools and utilities to help SAS Customers with SAS Viya running in containers.
Apache License 2.0
33 stars 13 forks source link

Pre Install Checker Reports Namespace Admin Permissions - Insufficient. #171

Closed lorenzk1213 closed 2 years ago

lorenzk1213 commented 2 years ago

Hi,

We have used viya4-iac to deploy the AWS environment. and used viya4-deployment DAC to deploy the baseline components, Right now, tried to run the viy4-ark pre-install checker, My Viya-Ark Pre-Install Checker Reports indequate permission for the Namespace Admin, Can pls help on how to address this issue?

Namespace Admin Insufficient Permissions

Thanks,

erharb commented 2 years ago

@lorenzk1213 these are permissions set up by whoever or whatever setup your kubernetes server. If you used IAC to create it then you may want to bring this up to them. If your kubernetes was created by someone else or your IT department then you''ll need to work with them to get your permissions corrected.

lasiva commented 2 years ago

@lorenzk1213 Please turn on the debug option and forward the logs to us. I'll take a look.

lorenzk1213 commented 2 years ago

Please find attached log with DEBUG enabled.

viya_pre_install_log_2022-05-26T06_27_46.log

*For security purposes, have deleted top portion of the log which contains AWS information

lasiva commented 2 years ago

@lorenzk1213 Thank you for the debug log.

  1. Please issue the following command from your kubectl client machine and share the output if possible. kubectl -n default api-resources -o wide I think you have permissions restrictions.
  2. Forward a copy of the report.
  3. Capture the command line options specified for the tool.
  4. The following failures are unexpected: image Though these resources were created successfully, the tool is not able to find them when it tries to delete them. Looking into it.

Thank you

lorenzk1213 commented 2 years ago

@lasiva Please find below output of kubectl -n default api-resources -o wide

api-resources.txt

Please find Pre-Install Report. As Im unable to upload using html, took screenshots to .docx

Pre Install Report.docx

Command line option used: python3 viya-ark.py pre-install-report -i nginx -H $INGRESS_HOST -p $INGRESS_HTTPS_PORT -d

lorenzk1213 commented 2 years ago

@lasiva I see the following 3 having Delete roles from the api-resources output. Could this be possibly a reporting tool bug?

roles rbac.authorization.k8s.io/v1 true Role [create delete deletecollection get list patch update watch]

rolebindings rbac.authorization.k8s.io/v1 true RoleBinding [create delete deletecollection get list patch update watch]

serviceaccounts sa v1 true ServiceAccount [create delete deletecollection get list patch update watch]

lasiva commented 2 years ago

The permissions seem to be available, but I see some issues .

  1. The end of the output from api-resources in your attached file shows the following: error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request.

This results in the tool capturing the following: Command 'kubectl -n default api-resources -o wide' returned non-zero exit status 1, The tool is interpreting that as insufficient Permissions. You might check with following command to make sure there are no issues on the cluster.
kubectl get pods -n kube-system Please Check if some pod are down.

  1. Also the failed deletes are unable to find the resources to delete. Like following.

"/home/925408/viya4-ark-1.8.0/pre_install_report/library/utils/viya-rolebinding.yaml": rolebindings.rbac.authorization.k8s.io "viyasrolebinding" not found\n' error_out b'' The tools is interpreting Not Found as Insufficient permissions which is misleading.

I'm not sure at this point why the tool is unable to find resources that it successfully created. Have not hit that before.

  1. Please provide the output of the "kubectl version" command also.
lasiva commented 2 years ago

@lorenzk1213 I'm attempting to reproduce your error. I'll run the tool against default namespace in a clean cluster.

In the mean time we think you could go ahead and try a full deployment and not let the pre-check tool block you.

fyi... we are closed for national holiday this Monday 30th. I may not be able to get back till late Monday or Tuesday,

Thank you

lasiva commented 2 years ago

@lorenzk1213 I was able to recreate your error scenario.

  1. Created an AWS cluster with no Viya deployment
  2. Deleted the metrics server pod
  3. Issued the kubectl api-services -n kube-system command and it resulted in the same error: "error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"
  4. Finally ran the pre_install_check tool and got the same results as the one supplied by you. The pre_install_check tool will fail some checks if the api-resources commands returns non-zero return code.
  5. Is there a metrics server installed and running? Let us know.

image @lasiva

lorenzk1213 commented 2 years ago

@lasiva Yes we do have metric server installed and running

image

lasiva commented 2 years ago

@lorenzk1213 The kubectl api-services -n kube-system command resulted in an error on you system: "error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request". The error was in your attachment attachment per my earlier comment. Please resolve that issue and try again. Though the metrics server is running, the command results in non-zero return code when you run the tool.

thpang commented 2 years ago

What helm chart is being used for the metrics servers? With k8s 1.19 and above it should be 5.11 and above : https://artifacthub.io/packages/helm/bitnami/metrics-server/5.11.0 This supports the metrics-server 0.6.0 and higher which is targeted to 1.19+

lorenzk1213 commented 2 years ago

@thpang

We are using version 5.11.7

image

lorenzk1213 commented 2 years ago

Could this be related to the metrics-server issue in https://github.com/sassoftware/viya4-deployment/issues/230 ?

thpang commented 2 years ago

If you code is depending on the metrics-server being installed in the cluster it would be a problem. That issue seems transient and is not related.

From: lorenzk1213 @.> Sent: Wednesday, June 1, 2022 6:10 AM To: sassoftware/viya4-ark @.> Cc: Thomas Pangborn @.>; Mention @.> Subject: Re: [sassoftware/viya4-ark] Pre Install Checker Reports Namespace Admin Permissions - Insufficient. (Issue #171)

EXTERNAL

Could this be related to the metrics-server issue in sassoftware/viya4-deployment#230https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsassoftware%2Fviya4-deployment%2Fissues%2F230&data=05%7C01%7Cthomas.pangborn%40sas.com%7Cc40a2699222a46aaf36208da43b6f929%7Cb1c14d5c362545b3a4309552373a0c2f%7C0%7C0%7C637896750389383338%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RwUwa6hXG5mnjQbYAKOsVkPqcqhZE2%2FEKS91PXdpikA%3D&reserved=0 ?

— Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsassoftware%2Fviya4-ark%2Fissues%2F171%23issuecomment-1143403740&data=05%7C01%7Cthomas.pangborn%40sas.com%7Cc40a2699222a46aaf36208da43b6f929%7Cb1c14d5c362545b3a4309552373a0c2f%7C0%7C0%7C637896750389383338%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=7KynYSBwKeAV602x1Yef%2FteNjDYV%2BRmZh4lOG1YdXrE%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAHYCFOVQGSWEG7MJMFW664DVM4ZJZANCNFSM5WX2JIWQ&data=05%7C01%7Cthomas.pangborn%40sas.com%7Cc40a2699222a46aaf36208da43b6f929%7Cb1c14d5c362545b3a4309552373a0c2f%7C0%7C0%7C637896750389383338%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=G%2F3nqiVQu0VvG58eh8GpNW%2BYfzqC9gAoQ%2B6bEObPfwM%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

lasiva commented 2 years ago

@thpang thanks for following up. We will look at the tool dependency on the metrics server.

lasiva commented 2 years ago

We dont have a dependency on the metrics server but tool is looking for a clean rc from the following command: kubectl -n default api-resources -o wide (which is failing on this issue due to metrics server).

Will investigate options further.

erharb commented 2 years ago

If kubectl returns a non-zero code we expect it means failure. I'd recommend entering an issue against kubectl if there's not already one entered for it.

erharb commented 2 years ago

I stand corrected, it turns out the use of this specific kubectl command as it relates to Viya4-ark is only to determine its return code in one particular if conditional, but not actually using the data returned by the command. @lasiva has determined that this conditional is not necessary for the intended purpose of the function it is in and can be safely removed without affecting the intended behavior.

kevinlinglesas commented 2 years ago

This issue is addressed in Release 1.8.1.