Determine root cause for permissions issues with using grader-setup-service image with tag

jgwerner commented 3 years ago

Overview

We created a Pull Request for the grader-setup-service to ensure the SQL operation is rolled back in case of an exception. However, after building and pushing the grader-setup-service from this PR the grader-setup-service encounters permissions issues when attempting to connect to other cluster resources.

Reproducing the error

Deploy the stack with the latest version of this repo's helm chart
Set the grader-setup-service using image illumidesk/grader-setup-service:582.
Stream logs for the grader-setup-service using kubectl logs -f grader-setup-service-<id> -n <namespace>

Summary of tests to date

Update the illumidesk/helm-chart to remove ClusterRoleBinding typos and duplicates
Deploy a hello world deployment

Establish a remote session with the grader-setup-service using the kubectl exec ... command:

kubectl exec --stdin --tty grader-setup-service-<id> -n <namespace> -- /bin/bash

Deploy a hello world deployment using the Python Kubernetes client with the following code in a test.py file (may be located in any directory, such as /tmp):

from kubernetes import client, config

def main():
    config.load_incluster_config()

    v1 = client.CoreV1Api()
    print("Listing pods with their IPs:")
    ret = v1.list_pod_for_all_namespaces(watch=False)
    for i in ret.items:
        print("%s\t%s\t%s" %
              (i.status.pod_ip, i.metadata.namespace, i.metadata.name))

if __name__ == '__main__':
    main()

Open a new terminal tab and stream logs from the grader-setup-service pod above:

kubectl logs -f grader-setup-service-<id> -n <namespace>

Change to the terminal tab that has the active bash session to the grader-setup-service pod. Then, run the hello world deployment by entering (where test.py represents the file that has the code from step 2 above):

python test.py

View results. If there is a permissions error, the stack trace should display something like so:

root@grader-setup-service-67699469fc-7vxmb:/illumidesk# python test.py
Listing pods with their IPs:
Traceback (most recent call last):
  File "test.py", line 16, in <module>
    main()
  File "test.py", line 9, in main
    ret = v1.list_pod_for_all_namespaces(watch=False)
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 16864, in list_pod_for_all_namespaces
    return self.list_pod_for_all_namespaces_with_http_info(**kwargs)  # noqa: E501
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 16967, in list_pod_for_all_namespaces_with_http_info
    return self.api_client.call_api(
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 239, in GET
    return self.request("GET", url,
  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 233, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '3eee613a-1bc4-4069-b4d2-4595253fd97a', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Tue, 15 Jun 2021 17:50:24 GMT', 'Content-Length': '284'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:gatech:illumidesk-account\" cannot list resource \"pods\" in API group \"\" at the cluster scope","reason":"Forbidden","details":{"kind":"pods"},"code":403}

Tests to date

To fix the permissions errors (403) mentioned above, the helm-chart was changed to:

Remove duplicate entries for the ClusterRole
Test by using Role/RoleBinding instead for ClusterRole/ClusterRoleBinding
Test by using ClusterRole with RoleBinding

Things that haven't been tested

Update the subject for the RoleBinding or ClusterRoleBinding to have something other than a service account as a subject, e.g. a user or group.
Replace the RoleBinding or ClusterRoleBinding after changing the role reference, e.g. updating the ClusterRole to a Role, etc.

...others

jgwerner commented 3 years ago

@Abhi94N to remove any possibility that there are issues with the Kubernetes Python client, let's test with the Go client that we need anyway from the new illumidesk/grader-setup-service. This folder has some examples that are similar to the Python client examples.

jgwerner commented 3 years ago

@Abhi94N closing. This was resolved with illumidesk/illumidesk#602.

IllumiDesk / helm-chart