Quay Enterprise on > Tectonic v1.4.5

[alekssaul]

Issue: When Quay Enterprise is deployed on Kubernetes, Quay is smart enough to detect the platform it's running on. On Kubernetes, Quay configuration is written to a secret object rather than a config.yaml file in the filesystem. With Tectonic v1.4.5 we have RBAC enabled in the Kubernetes API server and by default the serviceaccount for the namespace does not have access to update the Kubernetes Secret object. Quay being denied access to modify the secret causes failures within Quay to save it's configuration.

Proposed Solution: Create Role and RoleBindings for quay-enterprise namespace service account and grant it access to modify secret objects (quay-enterprise-config-secret ). Update the documentation to create these objects prior to Quay enterprise deployment.

quay-servicetoken-role.yaml

kind: Role
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
  namespace: quay-enterprise
  name: quay-enterprise-serviceaccount
rules:
  - apiGroups: [""] # The API group "" indicates the core API Group.
    resources: ["secrets"]
    verbs: ["get", "put", "patch", "update"]
    nonResourceURLs: []

quay-servicetoken-role-binding.yaml

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
  name: quay-enterprise-secret-writer
  namespace: quay-enterprise
subjects:
  - kind: ServiceAccount 
    name: default
roleRef:
  kind: Role
  namespace: quay-enterprise
  name: quay-enterprise-serviceaccount
  apiVersion: rbac.authorization.k8s.io/v1alpha1

I've tested the Role-RoleBinding above and seem to resolve the issue. I am hoping @ericchiang can take a look at this to make sure it's acceptable before I submit a PR. Role grants access to modify all secrets under "quay-enterprise" namespace, in reality we only need to modify quay-enterprise-config-secret object. I've also followed a naming convention that made sense to me, please provide feedback.

/cc: @josephschorr , @kbrwn as we talked about this.

[ericchiang]

This works for me.

[alekssaul]

customer had to add the get verb for namespace to the Role

kind: Role
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
  namespace: quay-enterprise
  name: quay-enterprise-serviceaccount
rules:
  - apiGroups: [""] # The API group "" indicates the core API Group.
    resources: ["secrets"]
    verbs: ["get", "put", "patch", "update"]
    nonResourceURLs: []
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get"]

[ericchiang]

Why's QE getting it's namespace? The namespace is in every container at:

/var/run/secrets/kubernetes.io/serviceaccount/namespace

[josephschorr]

@ericchiang: Not sure what you mean... it should read it from that mounted location?

[alekssaul]

I believe QE checks to make sure the namespace 'quay-enterprise' exists.

https://github.com/coreos-inc/quay/blob/2a7dbd3348e9c9adfa42433f50a81ee3e51d1ece/util/config/provider/k8sprovider.py#L77

[ericchiang]

@josephschorr you can just read that file and mandate the contents is "quay-enterprise". It holds the namespace your pod is running in. Would avoid the API call.

[ericchiang]

But yeah, the additional RBAC rule works too.

[josephschorr]

@ericchiang: Your @ didn't go to me :)

[alekssaul]

@ericchiang : Tested against 1.5.1 (**manual upgrade path). Role was transferable between K8s 1.4 to 1.5, however rolebinding had to be changed. Role Binding for Quay Enterprise Kubernetes 1.4 -https://gist.github.com/alekssaul/e29b915cf90e0a606ee0bb6476b79f94 vs. Role Binding for Quay Enterprise Kubernetes 1.5 - https://gist.github.com/alekssaul/596f633cc66ebb88a6fa90506c65e386

I believe we version control the documentation, so should be possible to document different manifests in Tectonic v1.4 vs v1.5. Hoping @joshix can keep me honest there. I believe I can submit a PR here to be synced for Tectonic 1.4.7, and merge a second PR for 1.5 later.

** Manual upgrade path tested - Upgrade API server ds to hyperkube 1.5.1, upgrade controller, scheduler and kubeproxy in order. confirm that kubectl version reported 1.5.1 for server.