feat: configurable kubectl cmd

[loeffel-io]

Hey, would love to test this plugin but i think there is something missing.

At my company we do hermetic builds with bazel where the build files living in a isolated directory (different for envs too).

With that being said, it would be amazing to customize the kubectl command.

for example we do run kubectl like this:

bazel run //deployments/dev:kubectl (kube config is stored deep in the file tree) bazel run //deployments/staging:kubectl (kube config is stored deep in the file tree)

thank you!

[mosheavni]

sounds reasonable. we have to assume that what you provide in the kubectl_cmd option to be 1:1 the same as regular kubectl, just a different path

[Ramilito]

ah, yeah we have assumed that kubectl is in path, a configuration to customise where to look for kubectl should solve this no?

[loeffel-io]

Great, just want to mention: it's not only the path, bazel run //deployments/dev:kubectl also downloads the kubectl cli itself somewhere into the tree to be hermetic.

so i think my issue would be solved if it would be possible to just configure the whole kubectl param like here: https://github.com/Ramilito/kubectl.nvim/blob/main/lua/kubectl/actions/kube.lua#L37

in my case my command would needs to be extended with --

for example: kubectl proxy must be bazel run //deployments/dev:kubectl -- proxy

[loeffel-io]

Just verified this

So i would need to configure the plugin to open it with bazel run //deployments/dev:kubectl -- for dev and bazel run //deployments/staging:kubectl -- for example

[loeffel-io]

maybe important: we do BUILD_USER=${BUILD_USER} bazel run //deployments/dev:kubectl -- params for dev, but i think i can configure the env outside the plugin (?, not sure how vim.system works)

not the easiest feature request for sure 😅

[mosheavni]

Just verified this

So i would need to configure the plugin to open it with bazel run //deployments/dev:kubectl -- for dev and bazel run //deployments/staging:kubectl -- for example

we should not implement a logic for different CLI for different clusters, this should be done by you using an autocmd that checks some logic and defines the config option for kubectl_cmd (which would need to be implemented)

[mosheavni]

@loeffel-io do you run every kubectl command with bazel? or just the first one to authenticate and then you're using a different cli?

[loeffel-io]

we run every kubectl command with bazel to be hermetic

some more insights: https://bazel.build/basics/hermeticity

[Ramilito]

It seems enough to offer a configuration for changing the kubectl part of the command that defaults to kubectl if not changed. Or maybe I'm missing something? That would solve this issue and also future issues where someone doesn't have kubectl in path.

Although, curious how it will work for subsequent calls to kubectl, would it need to download the binary again?

The environment variable should be fine since vim.system inherits them from current instance (unless cleared)

[loeffel-io]

Not sure if thats enough but i can test it for you ❤️

Bazel caches everything once downloaded and the executation is super fast

[loeffel-io]

maybe this is important: the bazel kubectl command also prints informations about the current google service account etc, maybe this affects your output "parsing" - not sure how you handle this

[loeffel-io]

bazel run //deployments/dev:kubectl_earth_user_service -- version                                                                                                                           +
INFO: Analyzed target //deployments/dev:kubectl_earth_user_service (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //deployments/dev:kubectl_earth_user_service up-to-date:
  bazel-bin/deployments/dev/kubectl_earth_user_service
INFO: Elapsed time: 0.112s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/deployments/dev/kubectl_earth_user_service external/com_github_mindful_hq_rules/pkg/gcloud/gcloud_lib.sh external/com_cloud_google_gcloud_cli_darwin_arm64/bin/gcloud earth earth-dev-382708 us-central1 earth-user-service-dev@earth-dev-382708.iam.gserviceaccount.com external/com_storage_googleapis_kubernetes_release_kubectl_darwin_arm64/file/kubectl dev version
WARNING: This command is using service account impersonation. All API calls will be executed as [earth-user-service-dev@earth-dev-382708.iam.gserviceaccount.com].
WARNING: This command is using service account impersonation. All API calls will be executed as [earth-user-service-dev@earth-dev-382708.iam.gserviceaccount.com].
WARNING: Accessing a Kubernetes Engine cluster requires the kubernetes commandline
client [kubectl]. To install, run
  $ gcloud components install kubectl <-- can be ignored

Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.7-gke.1274000

[loeffel-io]

I think these behavior is the same in makefile envs for example

[Ramilito]

Hmm that could be an issue, is there a silence flag? How would a kubectl get pod --output=json look like?

[loeffel-io]

I think i found a way easier solution

Hermeticity would be nice but is not super important to run kubectl.nvim. If kubectl.nvim offers to configure the --kubeconfig param on open it should work.

I then can pass the config via bazel run //deployments/dev:kubectl_earth_user_service -- config view

https://github.com/kubernetes/kubectl/issues/588

[loeffel-io]

But the other solution could also work and could be easier

If i do bazel run //deployments/dev:kubectl_earth_user_service -- version > test.txt the content of test.txt is only

Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.7-gke.1274000

[Ramilito]

That's a great suggestion as well, I think it's similar in work effort though.

Could you try to alias it? ''' alias kubectl="run bezel..." '''

and see what breaks?

[loeffel-io]

i think both config options (specifing the kubectl command and --kubeconfig option) would be great for kubectl.nvim. for myself i would prefer the hermetic way.

the alias would not work because the bazel commands differ from dev to staging and for every repository

bazel run //deployments/dev:kubectl_earth_user_service bazel run //deployments/staging:kubectl_earth_user_service bazel run //deployments/dev:kubectl_earth_resourcemanager_service

thats why it would be great to configure the above options on open

[Ramilito]

Yeah both are pretty important functionality as long as we use kubectl. Should be mostly a search replace effort!

[loeffel-io]

i tried this btw with a kubectl in path

return {
    {
        "ramilito/kubectl.nvim",
        config = function()
            require("kubectl").setup()
            vim.keymap.set("n", "<leader>k", function()
                vim.fn.system('bazel run //deployments/dev:kubectl_earth_user_service -- config view > test.yaml')
                vim.env.KUBECONFIG = "/Users/loeffel/go/src/github.com/mindful-hq/earth-user-service/test.yaml"

                -- Execute the original Lua command
                require("kubectl").toggle()
            end, { noremap = true, silent = true })
        end,
    },
}

but this results in no result (file is written great, the bazel part works) second time pressing the keystroke results in

E5108: Error executing lua: ...local/share/nvim/lazy/kubectl.nvim/lua/kubectl/state.lua:115: table index is nil
stack traceback:
        ...local/share/nvim/lazy/kubectl.nvim/lua/kubectl/state.lua:115: in function 'set_session'
        ....local/share/nvim/lazy/kubectl.nvim/lua/kubectl/init.lua:28: in function 'close'
        ....local/share/nvim/lazy/kubectl.nvim/lua/kubectl/init.lua:37: in function 'toggle'
        .../loeffel/.config/nvim/lua/loeffel-io/plugins/kubectl.lua:11: in function <.../loeffel/.config/nvim/lua/loeffel-io/plugins/kubectl.lua:6>

[loeffel-io]

passing the kubeconfig now as base64 string via toggle param would be just amazing (would be faster then running the bazel command everytime)

or passing the bazel command to toggle would be also amazing

[loeffel-io]

Just realised that kubeconfig will not work because kubectl depends on another google cloud hermetic files. this should be also the problem for the above error

this make sense because we call kubectl like this:

...

CLOUDSDK_CONFIG="$gcloud_config_path" \
KUBECONFIG="$kubectl_config_path" \
PATH=$gcloud_dir_path:$PATH \
  $kubectl \
  "${@:9}"

i think my only easy solution is to specify the custom kubectl command

[Ramilito]

I'll try to implement a configurable kubectl command later tonight when I have a bit more time!

[Ramilito]

Could you test using this branch: feat/configurable_kubectl_cmd?

I had to clear all environments when doing the calls since vim.system doesn't seem to replace duplicates which ended up with quite the unstable setup. If you could try with hardcoded values for now, we can try to make it dynamic on toggle later.

Example config:

kubectl_cmd = { env = { "KUBECONFIG=/Users/ramidaghlawi/.kube/test" }, cmd = "kubectl" }

Notice that I'm using full path and not expandable variables ☝️

[loeffel-io]

first of all: thank you ❤️

but:

same result with this config leader k first time: nothing leader k n times after: table error

is there any way to debug the output of the kubectl_cmd?

return {
    {
        "ramilito/kubectl.nvim",
        branch = "feat/configurable_kubectl_cmd",
        config = function()
            require("kubectl").setup({
                kubectl_cmd = {
                    -- env = { "KUBECONFIG=/Users/ramidaghlawi/.kube/test" },
                    cmd = "bazel run //deployments/dev:kubectl_earth_user_service",
                },
            })
            vim.keymap.set("n", "<leader>k", function()
                -- -- Get the current working directory
                -- local current_path = vim.fn.getcwd()
                --
                -- -- Print the current path
                -- print("Current path: " .. current_path)
                --
                -- vim.fn.system('bazel run //deployments/dev:kubectl_earth_user_service -- config view > ~/.kube/config')
                --
                -- if vim.v.shell_error == 0 then
                --     print("Command executed successfully")
                -- else
                --     print("Error executing command")
                -- end
                --
                require("kubectl").toggle()
            end, { noremap = true, silent = true })
        end,
    },
}

[loeffel-io]

same result with cmd = "bazel run //deployments/dev:kubectl_earth_user_service --",

[Ramilito]

I did miss to actually use the cmd that you changed 😊, could you try again?

[Ramilito]

is there any way to debug the output of the kubectl_cmd?

If it's an error it should print out the error but if you want to investigate a bit, you should look at the code in the lua/actions/kube.lua file, at row 41 is where we create the proxy (which I believe is our first issue)

[loeffel-io]

now getting this on first time and all other times

E5108: Error executing lua: ...lar/neovim/0.10.1/share/nvim/runtime/lua/vim/_system.lua:244: ENOENT: no such file or directory stack traceback: [C]: in function 'error' ...lar/neovim/0.10.1/share/nvim/runtime/lua/vim/_system.lua:244: in function 'spawn' ...lar/neovim/0.10.1/share/nvim/runtime/lua/vim/_system.lua:335: in function 'system' ...hare/nvim/lazy/kubectl.nvim/lua/kubectl/actions/kube.lua:41: in function 'start_kubectl_proxy' ....local/share/nvim/lazy/kubectl.nvim/lua/kubectl/init.lua:16: in function 'open' ....local/share/nvim/lazy/kubectl.nvim/lua/kubectl/init.lua:40: in function 'toggle' .../loeffel/.config/nvim/lua/loeffel-io/plugins/kubectl.lua:27: in function <.../loeffel/.config/nvim/lua/loeffel-io/plugins/kubectl.lua:12>

[Ramilito]

You get that error when it tries to execute something but can't find it, since we clear the environment variables in the vim.system shell, I already add the PATH and HOME envs back in but you might need to add other executables if they aren't in the PATH.

I'm guessing bazel isn't? or is it something else? you should be able to add it like this:

kubectl_cmd = {
    env = { "bazel=/Users/<USER>/bin/bazel" },
    cmd = "bazel run //deployments/dev:kubectl_earth_user_service",
}

[loeffel-io]

bazel is in the path 🤔

[Ramilito]

hmm, I set the clear_env to false, could you see if you get a step further? If that works, it would mean we need to add something to the env list, not sure what bazel depends on though.

[loeffel-io]

the same error, with -- and without --

[loeffel-io]

the weird thing is, using whoami for example works

[loeffel-io]

bazel without any params also work!

so i think there problem is how lua calls the cmd, because only bazel is cmd and the rest of bazel run //deployments/dev:kubectl_earth_user_service are args

[loeffel-io]

it feels like in this special case it would be good to also add args as option 🤔

[loeffel-io]

confirmation: putting the bazel cmd or any other cmd with args inside a bash script (test.sh) works, so the problem are the args

[loeffel-io]

with this

                kubectl_cmd = {
                    -- env = { "KUBECONFIG=/Users/ramidaghlawi/.kube/test" },
                    cmd = "/Users/loeffel/go/src/github.com/mindful-hq/earth-user-service/test.sh",
                },

and this bash script test.sh

#!/bin/bash

# loses access to my $BUILD_USER btw because i cant add this to the cmd and my global BUILD_USER is not available
BUILD_USER=loeffel-io bazel run //deployments/dev:kubectl_earth_user_service -- "$@"

it works, but bazel gets locked because two many processes are running, but thats maybe an issue with my long running process.

[loeffel-io]

just some more infos: bazel looses caching and must rebuild everytime running inside my test.sh, maybe thats why the envs are cleared. and currently i need to press my enter key after each logging line

sorry for all the spam

i really think this could work when i can add args to the cmd and call bazel directly

ref: https://github.com/bazelbuild/bazel/issues/11020

[Ramilito]

That's a great observation and explains the error, yeah args need to be a table (not everywhere but we handle that by table.concat). Added the args option as well to the config.

Something like this should work better:

kubectl_cmd = {
  env = { "BUILD_USER=loeffel-io" },
  cmd = "bazel",
  args = { "run", "//deployments/dev:kubectl_earth_user_service", " --", "$@" }
 },

Will keep the clear_env set to false until we have a successful call using bazel.

[loeffel-io]

Amazing! I am currently on vacation, will be back next week to test it

Thank you!

[Ramilito]

No worries! I'll merge it to main and we can adjust and reopen this issue if it didn't work!

[loeffel-io]

error: unable to load root certificates: unable to parse bytes as PEM block

this is what i get with the main branch

[Ramilito]

I see, I'm expecting you to have some environments variables missing tbh, could you try setting below to false? https://github.com/Ramilito/kubectl.nvim/blob/35de3b782c943e0e7e633411d33b2b4213acf7cd/lua/kubectl/actions/kube.lua#L41

It could also be something else but this one will at least rule env variable issues

[Ramilito]

Let me know if you are too busy to test it out, I can try setting up bazel in that case 🙏