Flatpak: cannot execute `aws eks get-token`

[gtirloni]

In order to access the Kubernetes API of an EKS cluster, it's necessary to use the aws eks get-token command.

This is usually configured in the kubeconfig file:

- name: eks_my-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - --region
      - us-east-1
      - eks
      - get-token
      - --cluster-name
      - my-cluster
      command: aws
      env:
      - name: AWS_PROFILE
        value: my-profile
      interactiveMode: IfAvailable
      provideClusterInfo: false

However, the Flatpak does not have access to the host's filesystem by default. This can be granted with filesystem=host but there are other issues.

The awscli executable is installed in /usr/bin/aws on the host but /usr/bin inside the Flatpak is different. To access the host's awscli one has to execute /var/run/host/usr/bin/aws, so adding /var/run/host/usr/bin to the PATH inside the Flatpak is necessary. But there are other issues.

The awscli tool is a Python application that loads other Python modules. The Python interpreter won't be able to find the host's dist-packages directory. It's necessary to add /var/run/host/usr/lib/python3/dist-packages to the PYTHONPATH environment variable inside the Flatpak. But there are other issues.

The awscli tool uses a native module called awscrt that is compiled for Python 3.11 but the Flatpak packages Python 3.10. When the interpreter tries to load awscli.checksums, it does not have permission to write to the __pycache__ directory:

$ python -v
>>> import awscrt.checksums
# code object from /var/run/host/usr/lib/python3/dist-packages/awscrt/__init__.py
# could not create '/var/run/host/usr/lib/python3/dist-packages/awscrt/__pycache__/__init__.cpython-310.pyc': PermissionError(13, 'Permission denied')

Setting PYTHONDONTWRITEBYTECODE=1 resolves the error but the module still can't be loaded (probably related to the binary .so that needs to be loaded).

It seems a better solution would be for Headlamp to pack all the tools required to authenticate to EKS/AKS/GKE into the Flatpak.

[joaquimrocha]

Hi @gtirloni , we don't support the exec part of the user in kubeconfig at the moment. So if authenticating requires running an external command, it won't work. We are planning on adding this functionality though.

[tsipo]

I have the same issue with the flatpak installation with both EKS and GKE clusters. 10:04:39.295 › server process stderr: 2023/12/15 10:04:39 http: proxy error: getting credentials: exec: executable gke-gcloud-auth-plugin not found 10:04:39.332 › server process stderr: 2023/12/15 10:04:39 http: proxy error: getting credentials: exec: executable aws not found I downloaded the deb package from the releases, this one works perfectly fine with the auth plugins. It's a well-known issue of Linux jailed apps installed using snap or flatpak, they cannot access anything out of their "jail". As the cloud providers' auth plugins are typically installed this way, it would be great if you standardize on non-jailed apps (like deb and rpm packages) i.s.o. flatpak.

[joaquimrocha]

@yolossn How is it possible that the eks auth seems to be reported working in the case of the deb? (I thought we don't do anything with the exec part of the kubeconfig)

@tsipo , we'll look into finding a solution for flatpak.

[yolossn]

@joaquimrocha We actually don't actively deal with the exec part, based on @tsipo comment I looked into it and to my surprise it worked in certain exec configurations and didn't work in others. On further investigation found that the kubernetes/client-go changes that we made is handling it internally for certain cases where interactive browser flow for authenticating is not required.

For example the kubelogin exec command used by AKS clusters support many methods like devicecode, interactive, spn, ropc, msi, azurecli, workloadidentity. The devicecode login method expects the user to open a link and paste the provided code and in case of azurecli the token fetch is handled by az. The kubernetes/client-go is internally able to handle non-interactive token fetching methods like the azurecli one.

[absmith82]

This seems to also affect the windows client as I cannot use the desktop app to login to my EKS cluster from the desktop, but the in-cluster service works fine.

[em-]

I have the same issue with DigitalOcean authentication.

@joaquimrocha is there already a github issue to track the general problem of exec-based authentication from the Flatpak?

I guess the possible approaches to avoid shipping all the possible cli tools involve using extensions, one for each tool, or use flatpak-spawn somehow to call out to the host.

[joaquimrocha]

@em- , I don't think we have an issue just for flatpak, but we should tackle that when we add support for the kubectl exec.

[joaquimrocha]

@gtirloni , I am trying to fix this. If you can help us test it, it'd be highly appreciated. See this comment for instructions.

[joebowbeer]

@joaquimrocha Let me know when I can help test. I need to connect to EKS clusters on macos. My kubeconfig entries use the client.authentication exec w/ aws eks get-token, like the one in the description.

[joaquimrocha]

I will prepare a build for everyone to try.

[joaquimrocha]

@joebowbeer and everyone, here is a build of Headlamp which should work with EKS. Please find the DMGs at the bottom of this action run (you need to be logged in to Github): https://github.com/headlamp-k8s/headlamp/actions/runs/10672802557

[joebowbeer]

@joaquimrocha Thanks! Your dmg works for me on macos, and so does release 0.25.1

Aside: I wish the cluster name column could be expanded, and custom names/avatars could be assigned to clusters, as my cluster names are very long and are chopped off in all the cluster views. However, if I hide all the columns except cluster name, then I can make out their full names.

[joaquimrocha]

Aside: I wish the cluster name column could be expanded, and custom names/avatars could be assigned to clusters, as my cluster names are very long and are chopped off in all the cluster views. However, if I hide all the columns except cluster name, then I can make out their full names.

It's a good idea. And we should maybe wrap those names, so all of them are visible.

[joebowbeer]

Aside: I wish the cluster name column could be expanded, and custom names/avatars could be assigned to clusters, as my cluster names are very long and are chopped off in all the cluster views. However, if I hide all the columns except cluster name, then I can make out their full names.

It's a good idea. And we should maybe wrap those names, so all of them are visible.

Sorry to hijack the discussion, but I wanted to clarify a couple things:

1. The Name column is showing the context name, which generally defaults to the cluster name, but can be overridden
2. Instead of hiding columns, another workaround is to assign shorter context names in the kubeconfig