search

microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.49k stars 2.61k forks source link

Kubernetes@V1 .KubectlOutput is wrong #14361

Closed richshadman closed 1 year ago

richshadman commented 3 years ago

Note

Issues in this repo are for tracking bugs, feature requests and questions for the tasks in this repo

For a list:
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks

If you have an issue or request for the Azure Pipelines service, use developer community instead:

https://developercommunity.visualstudio.com/spaces/21/index.html )

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
Type: Bug

Enter Task Name: Kubernetes@V1

list here (V# not needed):
https://github.com/microsoft/azure-pipelines-tasks/tree/master/Tasks/KubernetesV1

Environment

Azure Pipeline, private build agent, windows

Issue Description

The output from kubectl commands is malformed. Commas are randomly inserted throughout the output, and percent signs are replaced with '%25' (25 append).

Additionally outputs that are expected to be a single line have additional lines appended to them.

For now I am handling this via replace and trim in powershell tasks.

[Include task name(s), screenshots and any other relevant details]

Task logs

[Enable debug logging and please provide the zip file containing all the logs for a speedy resolution]

Troubleshooting

Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting

Error logs

[Insert error from the logs here for a quick overview]

shigupt202 commented 3 years ago

Can you please share the logs, and the command that you're trying to execute?

shigupt202 commented 3 years ago

Closing stale issue. Feel free to reopen if the problem still exists.

MichaelSayko commented 3 years ago

I encountered the same issue intermittently with Kubernetes@1. KubectlOutput returns the expected value, but with a leading comma or with a leading and training comma. For example, KubectlOutput returns ",blue," when the expected return value is "blue"

Server: Azure DevOps Server 2020 Agent: Private, Linux, 2.170.1

steps:

MichaelSayko commented 3 years ago

Below is the log from the latest occurrence of the issue.

The expected result of the kubect get command is "green"

The Kubectl task returned ",green" on the first and second attempts to execute the task. The Kubectl task returned "green" on the third attempt to execute the task.

First attempt:

2021-05-26T16:35:02.8470783Z ##[section]Starting: Get Live Ingress 2021-05-26T16:35:02.8679412Z ============================================================================== 2021-05-26T16:35:02.8680287Z Task : Kubectl 2021-05-26T16:35:02.8681039Z Description : Deploy, configure, update a Kubernetes cluster in Azure Container Service by running kubectl commands 2021-05-26T16:35:02.8681640Z Version : 1.169.2 2021-05-26T16:35:02.8682436Z Author : Microsoft Corporation 2021-05-26T16:35:02.8683633Z Help : https://aka.ms/azpipes-kubectl-tsg 2021-05-26T16:35:02.8684232Z ============================================================================== 2021-05-26T16:35:03.3394937Z Found tool in cache: kubectl 1.20.5 x64 2021-05-26T16:35:03.3405828Z Prepending PATH environment variable with directory: /apps/tfsagent7/_work/_tool/kubectl/1.20.5/x64 2021-05-26T16:35:03.9626455Z ============================================================================== 2021-05-26T16:35:03.9628577Z Kubectl Client Version: v1.20.5 2021-05-26T16:35:03.9638671Z Kubectl Server Version: v1.18.18+IKS 2021-05-26T16:35:03.9639603Z ============================================================================== 2021-05-26T16:35:04.8561671Z [command]/apps/tfsagent7/_work/_tool/kubectl/1.20.5/x64/kubectl get -n main ingress main -o jsonpath="{.metadata.annotations.color}" 2021-05-26T16:35:04.8563153Z commandOutput",green" 2021-05-26T16:35:04.8609414Z ##[section]Finishing: Get Live Ingress

Second attempt:

2021-05-26T16:35:02.8470783Z ##[section]Starting: Get Live Ingress 2021-05-26T16:35:02.8679412Z ============================================================================== 2021-05-26T16:35:02.8680287Z Task : Kubectl 2021-05-26T16:35:02.8681039Z Description : Deploy, configure, update a Kubernetes cluster in Azure Container Service by running kubectl commands 2021-05-26T16:35:02.8681640Z Version : 1.169.2 2021-05-26T16:35:02.8682436Z Author : Microsoft Corporation 2021-05-26T16:35:02.8683633Z Help : https://aka.ms/azpipes-kubectl-tsg 2021-05-26T16:35:02.8684232Z ============================================================================== 2021-05-26T16:35:03.3394937Z Found tool in cache: kubectl 1.20.5 x64 2021-05-26T16:35:03.3405828Z Prepending PATH environment variable with directory: /apps/tfsagent7/_work/_tool/kubectl/1.20.5/x64 2021-05-26T16:35:03.9626455Z ============================================================================== 2021-05-26T16:35:03.9628577Z Kubectl Client Version: v1.20.5 2021-05-26T16:35:03.9638671Z Kubectl Server Version: v1.18.18+IKS 2021-05-26T16:35:03.9639603Z ============================================================================== 2021-05-26T16:35:04.8561671Z [command]/apps/tfsagent7/_work/_tool/kubectl/1.20.5/x64/kubectl get -n main ingress main -o jsonpath="{.metadata.annotations.color}" 2021-05-26T16:35:04.8563153Z commandOutput",green" 2021-05-26T16:35:04.8609414Z ##[section]Finishing: Get Live Ingress

Third attempt:

2021-05-26T16:36:23.1018575Z ##[section]Starting: Get Live Ingress 2021-05-26T16:36:23.1229859Z ============================================================================== 2021-05-26T16:36:23.1230857Z Task : Kubectl 2021-05-26T16:36:23.1232005Z Description : Deploy, configure, update a Kubernetes cluster in Azure Container Service by running kubectl commands 2021-05-26T16:36:23.1232486Z Version : 1.169.2 2021-05-26T16:36:23.1233309Z Author : Microsoft Corporation 2021-05-26T16:36:23.1233980Z Help : https://aka.ms/azpipes-kubectl-tsg 2021-05-26T16:36:23.1234443Z ============================================================================== 2021-05-26T16:36:23.3932078Z Found tool in cache: kubectl 1.20.5 x64 2021-05-26T16:36:23.3942537Z Prepending PATH environment variable with directory: /apps/tfsagent7/_work/_tool/kubectl/1.20.5/x64 2021-05-26T16:36:23.8614298Z ============================================================================== 2021-05-26T16:36:23.8615733Z Kubectl Client Version: v1.20.5 2021-05-26T16:36:23.8616881Z Kubectl Server Version: v1.18.18+IKS 2021-05-26T16:36:23.8619026Z ============================================================================== 2021-05-26T16:36:24.7085198Z [command]/apps/tfsagent7/_work/_tool/kubectl/1.20.5/x64/kubectl get -n main ingress main -o jsonpath="{.metadata.annotations.color}" 2021-05-26T16:36:24.7086934Z commandOutput"green" 2021-05-26T16:36:24.7135241Z ##[section]Finishing: Get Live Ingress

pbcahill commented 3 years ago

@shigupt202 looks like we don't have a way to reopen this issue. Can you reopen please? Or should we open a new issue for this problem?

This problem started for us after we upgraded our on-premises Azure DevOps Server 2019.1 to 2020.01. The Kubectl task was working fine on v1.53.8, and is now experiencing the above behavior with the output on v1.169.2

shigupt202 commented 3 years ago

Reopening this issue.

shigupt202 commented 3 years ago

@pbcahill @MichaelSayko Can you try to use -o json along with the command and see if that works? I have figured out the actual cause of the problem. This line is actually trying to create a comma separated array of json's, but it's impacting other outputs as well. I'll check if changing this has any other implications and then raise a PR.

pbcahill commented 3 years ago

@shigupt202 thank you for the response and reopening. Glad you have figured out the cause for the problem and can try updating when able.

As for using "-o json" instead of "-o jsonpath", that didn't really work as the -o json returns the entire json response. The -o jsonpath filters the response to return just the value of the annotation we consume in a subsequent task.

shigupt202 commented 3 years ago

@pbcahill I meant use -o json before -o jsonpath. Your command should look something like this: kubectl get -n main ingress main -o json -o jsonpath="{.metadata.annotations.color}"

MichaelSayko commented 3 years ago

@shigupt202 I can run the following command from the kubectl command line and get the desired result.

$ kubectl get -n main ingress main -o json -o jsonpath="{.metadata.annotations.color}" green

However, I am not clear on how to modify the Kubectl task in the pipeline given that the task specifies an oututFormat of 'jsonpath="{.metadata.annotations.color}"'

steps:

richshadman commented 3 years ago

You should be able to change your output format to “none” and add the output formats to your arguments.

From: Michael Sayko @.> Sent: Friday, May 28, 2021 12:30 PM To: microsoft/azure-pipelines-tasks @.> Cc: Rich Shadman @.>; Author @.> Subject: Re: [microsoft/azure-pipelines-tasks] Kubernetes@V1 .KubectlOutput is wrong (#14361)

@shigupt202https://github.com/shigupt202 I can run the following command from the kubectl command line and get the desired result.

$ kubectl get -n main ingress main -o json -o jsonpath="{.metadata.annotations.color}" green

However, I am not clear on how to modify the Kubectl task in the pipeline given that the task specifies an oututFormat of 'jsonpath="{.metadata.annotations.color}"'

steps:

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/microsoft/azure-pipelines-tasks/issues/14361#issuecomment-850534074, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AOCLXF2TABARWSAWTUYWKM3TP7AHVANCNFSM4XJTPKAA.

shigupt202 commented 3 years ago

Give none in the outputFormat, and -o json -o jsonpath="{.metadata.annotations.color}" in the arguments.

richshadman commented 3 years ago

I meant to post a log to this ticket some time back - other priorities came up. We still see the commas however.

For this particular issue, I am using this task to automate execution a distributed jmeter load test across multiple pods simultaneously and capturing the output so we get quite a bit of verbose output and it is almost always riddled with commas.

Fortunately the output from jmeter does not really contain commas so our workaround was simply to strip the output of commas in an additional task (after capturing the output to a variable). That doesn't resolve the underlying issue of course - the commas are still there.

For other tasks that require more complex execution, or the need to pipe the result of kubectl to another script to manipulate the value, this task does not work. It would be nice to have something similar to a kubectl enabled script task where one has access to kubectl configured for a particular cluster with the option of executing in powershell, pscore or bash (depending on the agent) - similar to azure cli task. A bit off topic, however it could provide a more robust kubectl solution and possibly bypass the formatting issues altogether with this different approach.

inuvano-labs commented 3 years ago

We're experiencing the same issue as per the OP, even with @shigupt202 suggested fix.

This is what our pipeline task looks like:

      - task: Kubernetes@1
        displayName: "Get project configuration"
        name: projectConfigurationOutput
        inputs:
          connectionType: "Kubernetes Service Connection"
          namespace: "project"
          command: "get"
          arguments: "configmap core -o json -o jsonpath='{.data}'"
          outputFormat: 'none'

The task extracts all key/values from the configmap and outputs a json fragment which we can reference in subsequent tasks using env var ref $(projectConfigurationOutput.KubectlOutput).

However, randomly we see a "," prefixed to the output which breaks json parsing in the tasks.

We have to put an additional task in to strip any leading "," to be safe.

Any ideas when this can be looked at or suggest another workaround?

tao-zhang commented 3 years ago

Hi @shigupt202,

I'm seeing exactly the same issue, here is the Azure DevOps build output, even with -o json in arguments.

[debug] get

[debug] -n

[debug] dummy-tst

[debug] configmap

[debug] test-config

[debug] -o

[debug] json

[debug] -o

[debug] jsonpath='{.data}'

/usr/local/bin/kubectl get -n dummy-tst configmap test-config -o json -o jsonpath='{.data}'

[debug]Called Toolrunner exec() method asynchronously. Returning the promise.

[debug]Exit code 0 received from tool '/usr/local/bin/kubectl'

[debug]STDIO streams have closed for tool '/usr/local/bin/kubectl'

[debug]Toolrunner exec() method returned successfully for the kubectl command.

commandOutput'{"TEST":"11%11%"}'

[debug]set KubectlOutput='{"TEST":"11%2511%25"}'

[debug]Processed: ##vso[task.setvariable variable=KubectlOutput;issecret=false;]'{"TEST":"11%2511%25"}'

richshadman commented 2 years ago

@shigupt202,

I have now ran into this again for a different pipeline. This one I was simply outputting a list of deployments matching a label (see picture below). I now have to modify this new pipeline and add a script task to clean up the output so I can use it.

Any update on when this might be fixed?

Task:

        # Get deployments matching the label selector
        - task: Kubernetes@1
          name: GetDeployments
          displayName: 'Get Internal Deployment Manifests by label: ${{ parameters.DeployLabelSelector }}'
          inputs:
            command: get
            arguments: deploy -l "${{ parameters.DeployLabelSelector }}"
            outputFormat: name

Output:

image

krokofant commented 2 years ago

I think we're seeing this warning because of the faulty JSON mentioned in this issue:

##[warning]Capturing deployment metadata failed with error: YAMLException: unexpected end of the stream within a flow collection at line 1383, column 1:

I assume this code that is called fails to interpret it as JSON and then it fails on parsing it as YAML:

if (IsJsonString(res)) {
    parsedObject = JSON.parse(res);
}
else {
    parsedObject = yaml.safeLoad(res);
}
bormm commented 2 years ago

The issue is still happening. As anything is mentioned already, nothing more to add. Please fix.

holotrek commented 2 years ago

I am running into this. Sometimes it works. Other times it has a leading comma, trailing comma, or both. It would be great to have this fixed since using the Azure CLI means having to set up yet another service connection with access to the cluster/namespace and that is a pain.

Task:

  - task: Kubernetes@1
    name: GetK8sDeployments
    displayName: Get deployments for label ${{ parameters.webAppNamePrefix }}
    inputs:
      connectionType: 'Kubernetes Service Connection'
      kubernetesServiceEndpoint: ${{ parameters.serviceConnectionName }}
      namespace: 'env-${{ parameters.environment }}'
      command: 'get'
      arguments: |
        deployments -l app=${{ parameters.webAppNamePrefix }} -o=jsonpath='{.items[*].metadata.name}'
      secretType: 'dockerRegistry'
      containerRegistryType: 'Azure Container Registry'
      outputFormat: 'none'
martillodecampo commented 2 years ago

We're also seeing this. Very irritating, especially since it happens randomly. It would really be nice to see this fixed.

trdrake-tw commented 2 years ago

We too are seeing this, and I agree - it's very irritating.

tomaeda commented 1 year ago

Is this issue fixed? The latest Kubernetes@1 seems to fix it, but I can't confirm it because it occurs randomly. If so, please tell me the PR number.

rikat-ms commented 1 year ago

Hi @shigupt202, do you have any information whether this issue was resolved by #17267 or not? The similar issue #17536 was resolved by this PR (the task version is 1.214.0).

github-actions[bot] commented 1 year ago

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

shunkino commented 10 months ago

Though this issue is closed. Let me share my findings in case it might help someone in the future. First of all, from the perspective of how to use the Kubernetes@1 task, only the values json, yaml, and none are allowed for the outputFormat option in this task.

[Kubernetes@1 - Kubectl v1 task] https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/kubernetes-v1?view=azure-pipelines

#outputFormat: 'json' # 'json' | 'yaml' | 'none'. Output format. Default: json.

Therefore, use of notation to extract specific Json elements from the output, such as outputFormat: jsonpath="{.items}", might not be expected by the task. If none of the above options apply, this setting will be passed to the kubectl command as the -o option, so in most cases you will get the expected result with the above specification, but we need to understand that this task does not support such input (at least that's what document says).

That being said, here's what I found by digging into the source code: In the Kubernetes@1 Task, the places where the "commandOutput" message and the KubectlOutput Output Variable are set are in the following cleanup() function.

[azure-pipelines-tasks/Tasks/KubernetesV1/src/kubernetes.ts] https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/KubernetesV1/src/kubernetes.ts#L139

This function receives an array called result, which is set in commandImplementation.run, and the result of result.toString() is set to KubectlOutput output variable. In TypeScript, toString() on an array formats the string by separating each element with a comma, but if it contains an empty string, it does not skip that element, but represents it with an empty string plus a comma. Example: var result = ["'',"blue",""] console.log(result.toString()) => ",blue,"

From the above, I conjecture that the issue discussed in this issue is caused by the result variable containing an empty string element for some reason. Elements are added to the result variable array by an anonymous function that executes result.push(data), and this function is executed from the following run method.

[azure-pipelines-tasks/Tasks/KubernetesV1/src/kubernetescommand.ts] https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/KubernetesV1/src/kubernetescommand.ts#L11

In this function, the anonymous function is handled by the name outputUpdate, and it is executed at the timing when the callback function of command.on("stdout"... is executed (the value output by the stdout event of command is stored in the result variable). The stdout event of the above command is emitted when kubectl, which is launched as a child process, writes a string to the standard input and output.

[azure-pipelines-task-lib/node/toolrunner.ts] https://github.com/microsoft/azure-pipelines-task-lib/blob/master/node/toolrunner.ts#L1113C1-L1124C1

From these context, if for some reason, the event of cp.stdout?.on('data'... is called with an empty buffer data, it will result in the result variable containing an empty string element, and the behavior OP have pointed out will occur. I believe that this issue can be resolved by making the following modifications to filter out empty string from result array.

[azure-pipelines-tasks/Tasks/KubernetesV1/src/kubernetes.ts] https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/KubernetesV1/src/kubernetes.ts#L141C1-L142C52 Current implementation:

            console.log("commandOutput" + result);
            const resultString = result.toString();

Suggested implementation change:

            const resultString = result.filter(v => v).toString();
            console.log("commandOutput" + resultString);

By this change, even if the result variable contains an empty string element, they will not be reflected in resultString, and the KubectlOutput will also not contain an empty string element.

We hope this will be helpful.

ricardomomm commented 7 months ago

Please fix this, it is very annoying, I'm using this to keep the number of replicas of a service, so before a deploy I get the current number of replicas, store in a variable, and when deploying I set the replicas to that variable otherwise if I omit it will get it back to 1.

So sporadically my build fails because one command output returned ",1," instead of "1"

image