Charset mismatch in webhook body from Bitbucket Server

KalleOlaviNiemitalo commented 1 year ago

Jenkins and plugins versions report

Environment

```text Jenkins: 2.414.1 OS: Windows Server 2012 R2 - 6.3 Java: 17.0.8.1 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- Office-365-Connector:4.20.0 analysis-model-api:11.7.0 ansicolor:1.0.4 antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 atlassian-bitbucket-server-integration:3.6.0 authentication-tokens:1.53.v1c90fd9191a_b_ authorize-project:1.7.1 azure-ad:404.v99a_327a_b_8904 azure-sdk:132.v62b_48eb_6f32f basic-branch-build-strategies:81.v05e333931c7d bitbucket-server-checks:1.0-SNAPSHOT (private-49d94041-kalle) bootstrap5-api:5.3.0-1 bouncycastle-api:2.29 branch-api:2.1122.v09cb_8ea_8a_724 buildtriggerbadge:251.vdf6ef853f3f5 caffeine-api:3.1.8-133.v17b_1ff2e0599 checks-api:2.0.1 cloudbees-bitbucket-branch-source:832.v43175a_425ea_6 cloudbees-disk-usage-simple:182.v62ca_0c992a_f3 cloudbees-folder:6.848.ve3b_fd7839a_81 command-launcher:107.v773860566e2e commons-httpclient3-api:3.1-3 commons-lang3-api:3.13.0-62.v7d18e55f51e2 commons-text-api:1.10.0-78.v3e7b_ea_d5a_fe1 compress-artifacts:98.vb_20f3c77ddf7 configuration-as-code:1700.v6f448841296e copyartifact:722.v0662a_9b_e22a_c credentials:1271.v54b_1c2c6388a_ credentials-binding:631.v861c06d062b_4 data-tables-api:1.13.5-1 display-url-api:2.3.9 dtkit-api:3.0.2 durable-task:523.va_a_22cf15d5e0 echarts-api:5.4.0-5 extended-read-permission:53.v6499940139e5 fast-track:1.0.0 font-awesome-api:6.4.0-2 forensics-api:2.3.0 git:5.2.0 git-client:4.5.0 git-forensics:2.0.0 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 hudson-wsclean-plugin:1.0.8 instance-identity:173.va_37c494ec4e5 ionicons-api:56.v1b_1c8c49374e jackson2-api:2.15.2-350.v0c2f3f8fc595 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.8-1 jdk-tool:73.vddf737284550 jersey2-api:2.40-1 jira:3.11 job-restrictions:0.8 jquery3-api:3.7.0-1 junit:1217.v4297208a_a_b_ce lockable-resources:1185.v0c528656ce04 mailer:463.vedf8358e006b_ manage-permission:1.0.1 matrix-auth:3.2 matrix-project:808.v5a_b_5f56d6966 metrics:4.2.18-442.v02e107157925 mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_ mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_ okhttp-api:4.11.0-157.v6852a_a_fa_ec11 pipeline-build-step:505.v5f0844d8d126 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-graph-view:198.v0844db_6ca_554 pipeline-groovy-lib:687.v62591d623759 pipeline-input-step:477.v339683a_8d55e pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2144.v077a_d1928a_40 pipeline-model-definition:2.2144.v077a_d1928a_40 pipeline-model-extensions:2.2144.v077a_d1928a_40 pipeline-rest-api:2.33 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2144.v077a_d1928a_40 pipeline-stage-view:2.33 pipeline-utility-steps:2.16.0 plain-credentials:143.v1b_df8b_d3b_e48 plugin-util-api:3.3.0 prism-api:1.29.0-7 resource-disposer:0.23 scm-api:676.v886669a_199a_a_ script-security:1275.v23895f409fb_d sidebar-link:2.2.4 simple-queue:1.4.3 snakeyaml-api:2.2-111.vc6598e30cc65 ssh-credentials:308.ve4497b_ccd8f4 sshd:3.312.v1c601b_c83b_0e structs:325.vcb_307d2a_2782 timestamper:1.26 token-macro:384.vf35b_f26814ec trilead-api:2.84.v72119de229b_7 variant:60.v7290fc0eb_b_cd warnings-ng:10.4.0 workflow-aggregator:596.v8c21c963d92d workflow-api:1281.vca_5fddb_3fceb_ workflow-basic-steps:1042.ve7b_140c4a_e0c workflow-cps:3787.v8f5dcd14a_fa_c workflow-durable-task-step:1289.v4d3e7b_01546b_ workflow-job:1346.v180a_63f40267 workflow-multibranch:756.v891d88f2cd46 workflow-scm-step:415.v434365564324 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:865.v43e78cc44e0d ws-cleanup:0.45 xunit:3.1.3 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Windows Server 2012 R2

Reproduction steps

Have the Jenkins controller running on Windows. Create a repository and a "master" branch in Bitbucket Server. Create a multibranch pipeline project in Jenkins and connect it to the repository via the Bitbucket Branch Source plugin, enabling both branch discovery (all branches) and pull request discovery. Set up a webhook in Bitbucket Server so that it will notify Jenkins. Push to Bitbucket Server a branch "test-ä" with a commit that is not on the "master" branch. Push to Bitbucket Server a branch "test-z" with a different commit that is not on the "master" branch. Create a pull request from "test-ä" to "master". Create a pull request from "test-z" to "master".

Expected Results

Jenkins should create four jobs in the multibranch pipeline project, and build each of them:

for the branch "test-ä"
for the branch "test-z"
for the pull request from "test-ä" to "master"
for the pull request from "test-z" to "master"

Actual Results

Jenkins creates only two jobs in the multibranch pipeline project, and builds them:

for the branch "test-z"
for the pull request from "test-z" to "master"

The jobs related to branch "test-ä" are missing.

Then click "Scan Multibranch Pipeline Now" in Jenkins. That makes it create the jobs that were missing.

Anything else?

This seems to be a bug in how bitbucket-branch-source-plugin decodes the webhook request body to a string, here:

https://github.com/jenkinsci/bitbucket-branch-source-plugin/blob/43175a425ea6adab63393e6ab044f12b129871ba/src/main/java/com/cloudbees/jenkins/plugins/bitbucket/hooks/BitbucketSCMSourcePushHookReceiver.java#L80

IOUtils.toString(InputStream) uses the default charset, which is Windows-1252 on the Windows server where the Jenkins controller runs. The webhook request however has "Content-Type: application/json; charset=utf-8", so the charset does not match.

https://www.rfc-editor.org/rfc/rfc8259#section-8.1 requires JSON data to be encoded in UTF-8, so the plugin should just always use that.

$ git grep -F -w -e IOUtils.toString -- src/main
src/main/java/com/cloudbees/jenkins/plugins/bitbucket/client/BitbucketCloudApiClient.java:                String content = IOUtils.toString(response.getEntity().getContent());
src/main/java/com/cloudbees/jenkins/plugins/bitbucket/client/BitbucketCloudApiClient.java:            return IOUtils.toString(inputStream, StandardCharsets.UTF_8);
src/main/java/com/cloudbees/jenkins/plugins/bitbucket/hooks/BitbucketSCMSourcePushHookReceiver.java:        String body = IOUtils.toString(req.getInputStream());

The second call in BitbucketCloudApiClient was changed to specify UTF-8 in https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/78/commits/5dedcda5be68b1552973e8f21d776a60f378842e.

KalleOlaviNiemitalo commented 12 months ago

Verified that replacing IOUtils.toString(req.getInputStream()) with IOUtils.toString(req.getInputStream(), StandardCharsets.UTF_8) fixes the problem.

KalleOlaviNiemitalo commented 12 months ago

Probably, this IOUtils.toString call should likewise be changed to use StandardCharsets.UTF_8:

https://github.com/jenkinsci/bitbucket-branch-source-plugin/blob/55eb918e66f30ab81a86c9837c84dfc865586129/src/main/java/com/cloudbees/jenkins/plugins/bitbucket/client/BitbucketCloudApiClient.java#L950

However, I am not changing that in https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/763, because:

That code is not related to webhooks and thus does not belong to this issue.
That code is only executed if Bitbucket Cloud returns an HTTP status other than 200 OK. The plugin will then throw a BitbucketRequestException in any case. Whether the content in the exception message was correctly decoded does not seem very important.
I would not be able to test a fix.

jenkinsci / bitbucket-branch-source-plugin