helm within awx using wrong credentials

[howardjones]

SUMMARY

I have a playbook that uses community.kubernetes.k8s to create a namespace, then community.kubernetes.helm to deploy a chart into that namespace. I want to do this from awx. I've created a service-account in the k8s cluster to do this, and a credential in awx with the token/CA details from that service-account. I've attached that credential to the job template for the playbook, but helm appears to be using awx's own service account (awx), not the one I supplied (awx-deploy):

Error: list: failed to list: secrets is forbidden: User "system:serviceaccount:awx:awx" cannot list resource "secrets" in API group "" in the namespace "simple-deploy-hj30"

It's unclear from awx docs (and looking through the code) what it actually does with k8s credentials attached to a playbook.

Is this a supported configuration for the helm module?

ISSUE TYPE

• Bug Report

COMPONENT NAME

community.kubernetes.helm

ANSIBLE VERSION

  2.9.13

CONFIGURATION

OS / ENVIRONMENT

AWX 15.0.0 Ansible 2.9.13 latest

STEPS TO REPRODUCE

    - name: Deploy Helm Chart
      community.kubernetes.helm:
        binary_path: helm/linux-amd64/helm
        name: simple-{{ deployerCode }}
        chart_ref: helm/charts/simple_app
        release_namespace: simple-{{ deployerCode }}
        wait: true
        values:
          database:
            hostname: "{{ databaseIP }}"
            username: "{{ databaseUsername }}"
            password: "{{ databasePassword }}"
          app:
            replicas: 1

EXPECTED RESULTS

helm deploys chart

ACTUAL RESULTS

helm complains that a user (not the one expected) can't access secrets (which the expected user can)

Error: list: failed to list: secrets is forbidden: User "system:serviceaccount:awx:awx" cannot list resource "secrets" in API group "" in the namespace "simple-deploy-hj30"

[tima]

@howardjones Just to verify, did you install the community.kubernetes collection into AWX? If so, which version of the collection are you using.

[howardjones]

Yes, I have a collections/requirements.yml in my project repo. It calls for 1.0.0

[tima]

@howardjones: FWIW the latest version of that collection is 1.1.1. I doubt that will fix your issue, but I thought I'd just point that out for future reference.

You're also working with a pretty new addition to the AWX code base and it seems that the docs for it haven't caught up to it. Sorry about that.

I'm not an AWX developer, but from what I recall talking to those developers and I see in the code the K8s credential type uses ENV variables. I'm pretty sure AWX is passing in the K8s credentials, but they are not lining up with the what the helm module will pick up and recognize.

The helm module will pick up 'K8S_AUTH_CONTEXT' and 'K8S_AUTH_KUBECONFIG'.

From what I'm seeing in the commit that added that feature it is using this:

        'env': {
            'K8S_AUTH_HOST': '{{ host }}',
            'K8S_AUTH_API_KEY': '{{ bearer_token }}',
            'K8S_AUTH_VERIFY_SSL': '{{ verify_ssl }}',
            'K8S_AUTH_SSL_CA_CERT': '{{ tower.filename }}',
        },

So I know that suck and we have some work to do. Sorry about that oversight. I'm going to open a feature request ticket to address this in the helm modules in this collection.

The only work around I can think of at the moment is to use a kubeconfig file with the credentials for awx-deploy and provide that path to the helm module. One small add to 1.1 was the ability to read a vaulted kubeconfig file.

Hope that helps.

[howardjones]

Thanks for the in-depth response @tima! I started working on this about 2 weeks ago - hadn't seen the two new release of the collection - I've updated that now. So it seems that my solution for now would be a task to fake up a kubeconfig from the env vars? That's sounds workable.

[tima]

Closing. This is being tracked thru #279.