Help connecting to AKS cluster using exec

[PixelRobots]

We are trying to use the official C API for Kubernetes: https://github.com/kubernetes-client/c What we want is to connect to an AKS cluster from a local workstation, and use the API to list pods in a cluster. We have already installed kubectl/az/kubelogin/etc on this workstation, and listing pods via kubectl works just fine.

Our AKS is on version 1.24

We are following the example here.

However, after we load the K8s config and get the API client:

    int rc = load_kube_config(&basePath, &sslConfig, &apiKeys, NULL);   /* NULL means loading configuration from $HOME/.kube/config */
    if (rc != 0) {
        printf("Cannot load kubernetes configuration.\n");
        return -1;
    }
    apiClient_t *apiClient = apiClient_create_with_base_path(basePath, sslConfig, apiKeys);
    if (!apiClient) {
        printf("Cannot create a kubernetes client.\n");
        return -1;
    }
    list_pod(apiClient);

we receive a 401 error when we try to list pods:

    pod_list = CoreV1API_listNamespacedPod(apiClient, "default",    /*namespace */
    ....

Our Azure Infrastructure partner has suggested that this may be due to some authentication changes that were introduced recently. Can you please send us a sample c/c++ code that would work with AKS clusters version 1.24 or higher or point us into the right direction.

Many Thanks

[brendandburns]

The exec provider here:

https://github.com/kubernetes-client/c/blob/master/kubernetes/config/exec_provider.c

should work.

You need the kubelogin binary, and a config as described here:

https://github.com/Azure/kubelogin#web-browser-flow-default

If you have a config that uses exec and it works in kubectl but doesn't work in this client, that's a bug, please let us know about it.

[PixelRobots]

Thanks Brendan. I will pass the information on to the team and will update here after.

[ityuhui]

There is an example using the exec provider: https://github.com/kubernetes-client/c/tree/master/examples/exec_provider

Hope it helps

[shayan-eftekhari]

Hello @ityuhui ,

Thank you for providing us a sample. We tried to use this approach but we have trouble figuring out how to retrieve token. We tried to generate a token using the following command:

get-token --environment AzurePublicCloud --server-id --client-id --tenant-id --login devicecode (we also tried interactive)

Then we passed the generated token as the last argument to my_exec_provider.

The result is as the following:

kube_exec_and_get_result(): The buffer for exec args is not sufficient.
kubeconfig_exec(): The kubeconfig exec failed.
load_kube_config(): Cannot exec command in kubeconfig.
Cannot load kubernetes configuration.

We also tried to use refresh-token from ~/.kube/config but we receive HTTP 401.

Also, from my_exec_provider code, I see there is another option to use client private key and client certificate instead of token. May I ask how to retrieve the values for these two parameters?

[shayan-eftekhari]

Hello @brendandburns ,

Thank you for the suggestion you made yesterday. If I understood correctly, you suggested to use the web browser flow. We did that using the following kube config:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <our certificate-authority-data>
    server: [SERVER-ADDRESS]
  name: aks-aimms-dev
contexts:
- context:
    cluster: aks-aimms-dev
    user: clusterUser_rg-aks-aimms-dev_aks-aimms-dev
  name: aks-aimms-dev
current-context: aks-aimms-dev
kind: Config
preferences: {}
users:
- name: clusterUser_rg-aks-aimms-dev_aks-aimms-dev
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - get-token
      - --environment
      - AzurePublicCloud
      - --server-id
      - 6dae42f8-4368-4678-94ff-3960e28e3630
      - --client-id
      - 80faf920-1908-4b52-b5ef-a8e7bedfc67a
      - --tenant-id
      - b6b9252d-06ac-4ea3-8c02-f52b5c7dc792
      - --login
      - interactive
      command: kubelogin

We tried to load it using this: load_kube_config(&basePath, &sslConfig, &apiKeys, NULL);

The rest of the code is the same as the sample. This also result in HTTP 401.

[brendandburns]

Does this Kubeconfig file work correctly when you use kubectl?

[shayan-eftekhari]

Yes, kubectl works fine.

[ityuhui]

kube_exec_and_get_result(): The buffer for exec args is not sufficient.

This is a defect. The reason is insufficient buffer memory. Please enlarge the value: KUBECONFIG_EXEC_ARGS_BUFFER_SIZE and KUBECONFIG_EXEC_COMMAND_BUFFER_SIZE

https://github.com/kubernetes-client/c/blob/master/kubernetes/config/exec_provider.c#L8-L9

And take another try.

BTW, my_exec_provider is just an example of an exec binary that I think cloud platform providers should provide.

[brendandburns]

@shayan-eftekhari can you try increasing the buffer size per @ityuhui 's suggestion and see if that works with the exec provider config from kubelogin.

If that doesn't work, I will try to reproduce this locally and see what is happening.

[shayan-eftekhari]

Thank you for your suggestions @brendandburns and @ityuhui.

I increased the buffer size so I don't see the buffer size problem anymore when I use the generated token of kubelogin, yet I still get HTTP 401.

I tried these two approaches:

1. Using my_exec_provider and the following config file:

   apiVersion: v1
   clusters:
   - cluster:
   certificate-authority-data: <my certificate-authority-data>
   server: [SERVER-ADDRESS]
   name: kubernetes
   contexts:
   - context:
   cluster: kubernetes
   user: kubernetes-admin
   name: kubernetes-admin@kubernetes
   current-context: kubernetes-admin@kubernetes
   kind: Config
   preferences: {}
   users:
   - name: kubernetes-admin
   user:
   exec:
     command: "./my_exec_provider"
     apiVersion: "client.authentication.k8s.io/v1beta1"
     args:
     - "arg1"
     - "arg2"
     - "Token generated by `kubelogin get-token --environment AzurePublicCloud --server-id 6dae42f8-4368-4678-94ff-3960e28e3630 --client-id 80faf920-1908-4b52-b5ef-a8e7bedfc67a --tenant-id b6b9252d-06ac-4ea3-8c02-f52b5c7dc792 --login devicecode|azurecli|interactive`"

2. Using default kube config in ~/.kube/config which is:

   apiVersion: v1
   clusters:
   - cluster:
   certificate-authority-data: <my certificate-authority-data>
   server: [SERVER-ADDRESS]
   name: aks-aimms-dev
   contexts:
   - context:
   cluster: aks-aimms-dev
   user: clusterUser_rg-aks-aimms-dev_aks-aimms-dev
   name: aks-aimms-dev
   current-context: aks-aimms-dev
   kind: Config
   preferences: {}
   users:
   - name: clusterUser_rg-aks-aimms-dev_aks-aimms-dev
   user:
   exec:
     apiVersion: client.authentication.k8s.io/v1beta1
     args:
     - get-token
     - --login
     - azurecli
     - --server-id
     - 6dae42f8-4368-4678-94ff-3960e28e3630
     command: kubelogin
     env: null
     provideClusterInfo: false

[ityuhui]

For the first approach: Using my_exec_provider and the following config file Can you try with cURL to verify your token ?

curl -X GET $APISERVER/api --header "Authorization: Bearer $TOKEN" --insecure

[shayan-eftekhari]

Hi Hui Yu, Here is the output:

{ "kind": "APIVersions", "versions": [ "v1" ], "serverAddressByClientCIDRs": [ { "clientCIDR": "0.0.0.0/0", "serverAddress": "\"[SERVER-ADDRESS]" } ] }

The token looks fine.

[ityuhui]

If the token works fine, the exec provider example should work. Did you pass in your kubeconfig ? https://github.com/kubernetes-client/c/blob/859fc3f5eecda59f4eea6b07431aa2ebf38db779/examples/exec_provider/list_pod_by_exec_provider.c#L48

[shayan-eftekhari]

Hi Hui Yu,

Yes, I passed the custom kube config. Let me give you more update.

In my previous setup, I created a conan package for kubernetes and I was trying to connect to AKS in my project.

Now I am running a fresh Ubuntu 22.04 docker container. I increased buffer size to 4096 and I followed the instructions listed here: https://github.com/kubernetes-client/c to compile kubernetes and exec_provider example.

Then I used the following command to generate the token: kubelogin get-token --environment AzurePublicCloud --server-id 6dae42f8-4368-4678-94ff-3960e28e3630 --client-id 80faf920-1908-4b52-b5ef-a8e7bedfc67a --tenant-id b6b9252d-06ac-4ea3-8c02-f52b5c7dc792 --login devicecode

Then I modified the config_with_exec_provider like this:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: [MY CERTIFICATE AUTHORITY DATA]
    server: [SERVER-ADDRESS]
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    exec:
      command: "./my_exec_provider_bin"
      apiVersion: "client.authentication.k8s.io/v1beta1"
      env:
      - name: "exec_client_certificate_data"
        value: "-----BEGIN CERTIFICATE-----\n\n-----END CERTIFICATE-----"
      - name: "exec_client_private_key"
        value: "-----BEGIN RSA PRIVATE KEY-----\n\n-----END RSA PRIVATE KEY-----"
      args:
      - "arg1"
      - "arg2"
      - "[GENERATED TOKEN]"

Everything else is exactly the same as the sample. Here is the output:

The return code of HTTP request=401 Cannot get any pod.

[ityuhui]

Let's go back to the 2nd approach: Using default kube config in ~/.kube/config which is:

Can you update the command with an absolute path in your ~/.kube/config

      command: /path/to/kubelogin

And try to debug the token getting from kubelogin at https://github.com/kubernetes-client/c/blob/859fc3f5eecda59f4eea6b07431aa2ebf38db779/kubernetes/config/exec_provider.c#L90

You can enable debugging when buiding the c client library:

mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=/usr/local ..
make
sudo make install

[shayan-eftekhari]

Hi Hui Yu,

Thank you Hui Yu for your suggestion.

We logged the token in exec_provider and it was fine. Then we followed the code logic until we found this function:

https://github.com/kubernetes-client/c/blob/master/kubernetes/config/kube_config.c#L86

This function truncates the token because BEARER_TOKEN_BUFFER_SIZE is only 2048 bytes.

We increased the buffer size here as well and the problem is solved.

You may want to increase this buffer size as well as those two you mentioned earlier in the library.

Thank you for all your help, Shayan

[brendandburns]

Glad you got this fixed, we should update the code to have a larger buffer size (and we should probably also print better error messages if the token is too long :)

[ityuhui]

Yes. the errors of insufficient buffer are caught and printed in the function kube_exec_and_get_result, but not in setApiKeys. I'm going to add this.