external auth (exec) not working (azure/kubelogin and aws-iam-authenticator)

[hofq]

I tried connecting to our prod clusters using "azure/kubelogin" and aws-iam-authenticator.

None of these Auth Methods worked.

Is there an eta we can expect exec auth to work? We want to switch from the closed source software lens to headlamp, but this is currently blocking.

[illume]

Hi.

Thanks for writing.

Yeah, we use the k8s APIs rather than kubectl and haven't implemented support for the "client-go credential plugins".

As a not-great work around authentication for both AKS and EKS can be done with OIDC. I realize this is probably not suitable however. Because the point of the aws-iam-authenticator is to avoid having to manage separate services. (please correct me if I'm wrong?)

Supporting external auth commands isn't currently on the short term roadmap. But I personally think it's something we should support.

---

External auth related docs

• client-go credential plugins
• aws-iam-authenticator
• azure/kubelogin

exec:
    apiVersion: ...
    command: ...
    args: ...

Other related issues:

A number of others have reported here and elsewhere wanting support for external auth commands.

Here are some of the github issues:

• 1582

• 1341

[gaeljw]

I agree this will be a must have for most companies.

[strowi]

Would also like for this to work. I'm guessing there are a lot of people using some form of client-side addon (e.g. i am using passman to store the credentials in keychain):

- name: rancher
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - keychain
      - user@cluster
      command: kubectl-passman
      env: null
      interactiveMode: IfAvailable
      provideClusterInfo: false

[lindblombr]

From #1716 I've been able to get this working like so

Can confirm what @kdeyko says. I have a similar issue with a custom OIDC binary we invoke from exec. If we open the app through point-and-click (macOS), we just get bad gateway and lots of errors in the dev console connecting to localhost. If I open up in a terminal via /Applications/Headlamp.app/Contents/MacOS/Headlamp, everything works perfectly.

Could be a work-around until the underlying connectivity issue is resolved?

[lindblombr]

On macos, I've used the following work-around to make this work via native Finder/Dock/Spotlight launch

$ sudo launchctl config user path /bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/homebrew/bin

This ensures that the paths where my exec plugins are located are in the environment of every application I launch via Finder et al. Unfortunately, the change requires root and a reboot, but successfully unblocks Headlamp in local GUI mode to work properly with exec configuration.

Incorporating something similar to shell-path in the electron startup should allow Headlamp to inherit user-configured PATH environment so that Headlamp works as expected: https://github.com/sindresorhus/shell-path (hopefully there is a newer, maintained dependency). Headlamp appears to implement kubeconfig handling in a way that already supports exec so long as any executable referenced by exec is either absolute path or IN the search PATH. TL;DR, enabling this feature seems to only require fixing the way Headlamp launches with respect to incorporating/honoring user-configured PATH that tools like kubectl already depend on.

[illume]

Thanks @lindblombr

[pg185210]

Hi.

Thanks for writing.

Yeah, we use the k8s APIs rather than kubectl and haven't implemented support for the "client-go credential plugins".

As a not-great work around authentication for both AKS and EKS can be done with OIDC. I realize this is probably not suitable however. Because the point of the aws-iam-authenticator is to avoid having to manage separate services. (please correct me if I'm wrong?)

Supporting external auth commands isn't currently on the short term roadmap. But I personally think it's something we should support.

External auth related docs

• client-go credential plugins
• aws-iam-authenticator
• azure/kubelogin

exec:
    apiVersion: ...
    command: ...
    args: ...

Other related issues:

A number of others have reported here and elsewhere wanting support for external auth commands.

Here are some of the github issues:

• Flatpak: cannot execute aws eks get-token #1582
• bad gateway while accessing AKS cluster #1341

Yep, I'd love that. OpenLens picks up my AWS EKS context and connects to the clusters out of the box. Headlamp recognizes the contexts, but just shows "Bad Gateway" status for the clusters. I guess I'll stay with OpenLens for now :(

edit2: Oh wait, yes, looks like it's exactly the problem:

16:18:10.725 › server process stderr: 2024/05/08 16:18:10 http: proxy error: getting credentials: exec: executable aws not found

It looks like you are trying to use a client-go credential plugin that is not installed.

To learn more about this feature, consult the documentation available at:
      https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins

thats what my kube config looks like:

- name: arn:aws:eks:us-west-2:011111111111:cluster/mycluster1
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - --region
      - us-west-2
      - eks
      - get-token
      - --cluster-name
      - mycluster1
      - --output
      - json
      command: aws

[joebowbeer]

FWIW, octant also implements client-go authentication.

[tstoeckler]

Facing this issue with the Linux Flatpak, as well. I need to connect to DigitalOcean clusters, so my particular auth binary of choice is doctl, but the Linux Flatpak does not see that. With the AppImage it works fine, but was wondering if this could be fixed for the Flatpak as well.

Not sure if it's feasible to actually package the needed binaries in the flatpak or rather to somehow expose binaries from the host.

[illume]

I think this fix might solve things for some people:

• https://github.com/headlamp-k8s/headlamp/pull/2172

A release is coming soon, but here are some release candidate builds. If anyone wants to test it before the release, I'd much appreciate it.

LINUX: https://github.com/headlamp-k8s/headlamp/actions/runs/10084955106 MAC: https://github.com/headlamp-k8s/headlamp/actions/runs/10084953201 WINDOWS: https://github.com/headlamp-k8s/headlamp/actions/runs/10084951177

(Moving the milestone to the next release milestone)

[joaquimrocha]

Facing this issue with the Linux Flatpak, as well. I need to connect to DigitalOcean clusters, so my particular auth binary of choice is doctl, but the Linux Flatpak does not see that. With the AppImage it works fine, but was wondering if this could be fixed for the Flatpak as well.

Not sure if it's feasible to actually package the needed binaries in the flatpak or rather to somehow expose binaries from the host.

For flatpak, I wonder if giving access (through Flatseal) to the locations that are needed (where you have your doctl binary) is enough?

[em-]

The binaries would also need to somehow end up in the $PATH used inside the Flatpak, I don't think it can be customized outside of the app itself.

And that's assuming that the binaries are statically compiled and there's no clash with the libraries inside the Flatpak.

As I naively see it, either:

1. the Headlamp Flatpak ships a bunch of extensions for each provider authentication tool (doctl, kubelogin, etc.)
2. Headlamp somehow forces the authentication backend to prefix each authentication command with flatpak-spawn --host

I have no idea how easy each of those is, or if they are feasible at all with reasonable effort, but at the moment I cannot use the Flatpak'ed Headlamp at all since my Azure clusters have now moved to the kubelogin auth helper tool (and the DigitalOcean ones never worked to begin with).

[joaquimrocha]

@em- , we are aware of this limitation for flatpak. I think the most feasible thing is to try approach 2. I do think that users can still tweak it from outside now, though not very conveniently: override the PATH env var (and any other needed like LD_LIBRARY_PATH, but I think probably just PATH will work with most tools) to include /run/host/bin; and change the permissions to include host. Flatseal can help do this, or one can use something like (assuming the needed files are installed in the host): flatpak override io.kinvolk.Headlamp --env=PATH=/run/host/bin:/usr/bin:/usr/local/bin:/app/bin --filesystem=host But I understand this may still need many more tweaks for things to work. We should definitely try approach 2.

[joaquimrocha]

@em- , I have created this PR which makes wrapper scripts for some common k8s tools, so they are run with flatpak-spawn. It also gives access to home and host, should it be needed. Can you try install the version that the CI will create (a comment will appear in the PR with the commands, if the build succeeds): https://github.com/flathub/io.kinvolk.Headlamp/pull/39

I am not sure this is all that's needed for e.g. the aws command to work, as the team is looking into another issue with it, but I think this one should at least make the tools visible to the Headlamp flatpak.

[joaquimrocha]

@em- , so, I added wrappers to be able to call external tools but the flathub maintainers have told me that I should build these apps into the bundle rather than asking for an exception for the linter to let the ability to call external apps pass. For now, I would like to understand if this PR works for flatpak users whose kubeconfigs need to run external tools like aws or kubelogin, etc..

2 steps are needed for testing this:

1. Install this test version with the instructions at: https://github.com/flathub/io.kinvolk.Headlamp/pull/39#issuecomment-2307375287
2. Override Headlamp's flatpak options to run with --talk-name=org.freedesktop.Flatpak. This can be done by running: flatpak override --talk-name=org.freedesktop.Flatpak io.kinvolk.Headlamp or by using the Flatseal app (also available in Flathub).

[joaquimrocha]

BTW, if environment variables are needed (i.e. kubeconfig has env vars set in the exec part), it may not easily work with flatpak. So you may need to pass those variables to flatpak, or, maybe a much better idea, set those env vars as config in .aws.

[Jeff-SearchPilot]

Just retried with the latest release and.. its working! Thanks!

Fedora 40 0.25.1 AppImage AWS EKS

[siegenthalerroger]

A release is coming soon, but here are some release candidate builds. If anyone wants to test it before the release, I'd much appreciate it.

[...]

MAC: https://github.com/headlamp-k8s/headlamp/actions/runs/10084953201

@illume the artifacts have expired, do you think you could trigger a new build? I'd love to test if this works with azure/kubelogin now.

[illume]

Hi @siegenthalerroger,

the latest release contains the fixes. Use those and hopefully it should be good.

[Cristobal-M]

Hi I am testing Headlamp with EKS for the first time and I've got this error when attempting to connect with my cluster

http: proxy error: getting credentials: exec: fork/exec /app/bin/aws: exec format error

I manually edited the script file like this

#!/bin/bash
command="aws $@"
flatpak-spawn --host bash --login -c "$command"

And is working properly from a shell and the app

If I remove "#!/bin/bash" it fails again with "exec format error". I'm using Ubuntu 24.04

---

Also, I first tried with:

#!/bin/bash
flatpak-spawn --host bash --login -c "aws $@"

But I got this error

I don't know why. Is there something weird with string interpolation?

[limbik-martinj]

I am also seeing the same issue:

server process stderr: 2024/09/28 13:32:32 http: proxy error: exec: fork/exec /app/bin/aws: exec format error

Running latest flatpak version: Version: 0.25.1

Fedora 40

[joaquimrocha]

Hi I am testing Headlamp with EKS for the first time and I've got this error when attempting to connect with my cluster

http: proxy error: getting credentials: exec: fork/exec /app/bin/aws: exec format error

I manually edited the script file like this

#!/bin/bash
command="aws $@"
flatpak-spawn --host bash --login -c "$command"

And is working properly from a shell and the app

I hadn't seen this! Thanks a lot @Cristobal-M , I will try to get those updated on our side too.

[joaquimrocha]

I am also seeing the same issue:

server process stderr: 2024/09/28 13:32:32 http: proxy error: exec: fork/exec /app/bin/aws: exec format error

Running latest flatpak version: Version: 0.25.1

Fedora 40

Okay. Seems like I really need to update the scripts. I believe I can do it this week.

[joaquimrocha]

Hi @Cristobal-M . I have added your suggested changes. Can you test this build of flatpak? https://github.com/flathub/io.kinvolk.Headlamp/pull/41#issuecomment-2391090181

Please let me know if it works so I merge it.

[em-]

I just managed to test the flatpak for 0.25.1 and after some small adjustments it works very nicely, thank you so much! I can now access my new clusters with Headlamp, yay! :D

For what is worth, I first had to pass --talk-name=org.freedesktop.Flatpak or it would lack permissions to use flatpak-spawn --host (I guess I will now customize it with Flatseal):

server process stderr: Portal call failed: org.freedesktop.DBus.Error.ServiceUnknown
Hint: --host only works when the Flatpak is allowed to talk to org.freedesktop.Flatpak

And since this is a graphical application it did not initially get the customized PATH I used in my interactive shells, so I had to fix that in my bash configuration files.

Shall anyone have to debug this as I did:

$ flatpak run --talk-name=org.freedesktop.Flatpak io.kinvolk.Headlamp &
$ flatpak enter io.kinvolk.Headlamp bash
bash-5.2$ flatpak-spawn --host bash --login -c 'which kubelogin'

[Cristobal-M]

Hi @Cristobal-M . I have added your suggested changes. Can you test this build of flatpak? flathub/io.kinvolk.Headlamp#41 (comment)

Please let me know if it works so I merge it.

Sorry, I have been a bit disconnected in the last days It returns 404, I see it has been merged into master

[joaquimrocha]

Hi @Cristobal-M . I have added your suggested changes. Can you test this build of flatpak? flathub/io.kinvolk.Headlamp#41 (comment) Please let me know if it works so I merge it.

Sorry, I have been a bit disconnected in the last days It returns 404, I see it has been merged into master

No worries. It's been merged. So you should be able to update to test the new changes.

[tstoeckler]

This also works for me beautifully now, i.e. I can connect to my DigitalOcean clusters which use the local doctl binary. I used the sudo flatpak override --talk-name=org.freedesktop.Flatpak io.kinvolk.Headlamp as mentioned above so I can just start the flatpak from the GUI and have everything work as it should. Thanks again, that's really awesome and a game changer for me!