Renovate fails create PullRequest on Bitbucket Server (when special characters are in the chagelog?)

[AB-xdev]

Similar to #6424

Background:

We use Renovate in Docker (renovate/renovate:24-slim) and set RENOVATE_UNICODE_EMOJI to false. This also get's detected by Renovate as I see in the logs (Loglevel=Debug):

DEBUG: Combined config
       "config": {
         "hostRules": [
           {"hostType": "github", "domainName": "github.com", "token": "***********"}
         ],
         "binarySource": "docker",
         "platform": "bitbucket-server",
         "endpoint": "https://bitbucket.xxx.com",
         "username": "xxx@xxx.de",
         "password": "***********",
         "gitAuthor": "Renovate Bot <xxx@xxx.de>",
         "unicodeEmoji": false,
         "repositories": ["xxx/xxx"]
       }

The targeted repo - where the problem occurs - is on a Bitbucket Server (7.6) which uses MySQL (5.7) in the background.

The problem

The creation of the PR for org.testcontainers (1.15.2) always fails silently (no bad exit code, no warning message):

    DEBUG: Pull request creation error (repository=xxx/xxx, branch=renovate/org.testcontainers.version)
           "err": {
             "name": "HTTPError",
             "timings": {
               "start": 1616158210567,
               "socket": 1616158210568,
               "lookup": 1616158210568,
               "connect": 1616158210569,
               "secureConnect": 1616158210570,
               "upload": 1616158210570,
               "response": 1616158210650,
               "end": 1616158210651,
               "phases": {
                 "wait": 1,
                 "dns": 0,
                 "tcp": 1,
                 "tls": 1,
                 "request": 0,
                 "firstByte": 80,
                 "download": 1,
                 "total": 84
               }
             },
             "message": "Response code 500 (Internal Server Error)",
             "stack": "HTTPError: Response code 500 (Internal Server Error)\n    at Request.<anonymous> (/usr/src/app/node_modules/got/dist/source/as-promise/index.js:117:42)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)",
             "options": {
               "headers": {
                 "user-agent": "https://github.com/renovatebot/renovate",
                 "accept-encoding": "gzip, deflate",
                 "x-atlassian-token": "no-check",
                 "authorization": "***********",
                 "accept": "application/json",
                 "content-type": "application/json",
                 "content-length": "15355"
               },
               "url": "https://bitbucket.xxx.com/rest/api/1.0/projects/xxx/repos/xxx/pull-requests",
               "username": "",
               "password": "",
               "method": "POST",
               "http2": false
             },
             "response": {
               "statusCode": 500,
               "statusMessage": "Internal Server Error",
               "body": {
                 "errors": [
                   {
                     "context": null,
                     "message": "A database error has occurred.",
                     "exceptionName": "com.atlassian.bitbucket.DataStoreException"
                   }
                 ]
               },
               "headers": {
                 "server": "nginx/1.14.0 (Ubuntu)",
                 "date": "Fri, 19 Mar 2021 12:50:10 GMT",
                 "content-type": "application/json;charset=UTF-8",
                 "transfer-encoding": "chunked",
                 "connection": "close",
                 "x-arequestid": "@FEWDHWx770x222259x0",
                 "x-auserid": "1603",
                 "x-ausername": "xxxx",
                 "cache-control": "no-cache, no-transform",
                 "vary": "accept-encoding,x-auserid,cookie,x-ausername,accept-encoding",
                 "x-content-type-options": "nosniff",
                 "content-encoding": "gzip"
               },
               "httpVersion": "1.1"
             }
           }

So basically a branch get's created, but the PR is missing.

I managed to extract the data which is sent to bitbucket by setting the LOG_LEVEL to TRACE:

TRACE: Applying Basic authentication (repository=xxx/xxx, branch=renovate/org.testcontainers.version)
       "url": "https://bitbucket.xxx.com/./rest/api/1.0/projects/xxx/repos/xxx/pull-requests"
TRACE: got request (repository=xxx/xxx, branch=renovate/org.testcontainers.version)
       "url": "https://bitbucket.xxx.com/./rest/api/1.0/projects/xxx/repos/xxx/pull-requests",
       "options": {
         "method": "post",
         "hostType": "bitbucket-server",
         "baseUrl": "https://bitbucket.xxx.com/",
         "json": {
           "title": "Update org.testcontainers.version to v1.15.2",
           "description": "This PR contains the following updates:\n\n| Package | Type | Update | Change |\n|---|---|---|---|\n| [org.testcontainers:selenium](https://testcontainers.org) ([source](https://github.com/testcontainers/testcontainers-java)) | test | patch | `1.15.1` -> `1.15.2` |\n| [org.testcontainers:mysql](https://testcontainers.org) ([source](https://github.com/testcontainers/testcontainers-java)) | test | patch | `1.15.1` -> `1.15.2` |\n| [org.testcontainers:testcontainers](https://testcontainers.org) ([source](https://github.com/testcontainers/testcontainers-java)) | test | patch | `1.15.1` -> `1.15.2` |\n\n---\n\n### Release Notes\n\n\n**testcontainers/testcontainers-java**\n\n### [`v1.15.2`](https://github.com/testcontainers/testcontainers-java/releases/1.15.2)\n\n[Compare Source](https://github.com/testcontainers/testcontainers-java/compare/1.15.1...1.15.2)\n\n##### What's Changed\n\n-   What `1984` means to you? To us, this number means PR [#​1984](https://github.com/testcontainers/testcontainers-java/issues/1984), one of the oldest PRs we had open and... finally merged! 😅 Thanks to an amazing contribution by [@​seglo](https://github.com/seglo), we now provide an example of testing Kafka clusters where multiple `KafkaContainer`s are connected into one network. Try it!\n-   Another \"old\" PR is [#​3180](https://github.com/testcontainers/testcontainers-java/issues/3180) by [@​oussamabadr](https://github.com/oussamabadr). Those of you who run Selenium tests with Testcontainers will appreciate this newly added option to use (scrollable!) MP4 format instead of FLV.\n-   The connection with Ryuk (our watchdog sidecar container) now sets the socket timeout and retries the failures - helps with some rare networking edge cases. ([#​3682](https://github.com/testcontainers/testcontainers-java/issues/3682)) [@​diegolovison](https://github.com/diegolovison)\n-   The logs consumer no longer adds extra new lines thanks to [#​3752](https://github.com/testcontainers/testcontainers-java/issues/3752) by [@​perlun](https://github.com/perlun)\n-   Locally built images no longer get affected by the `hub.image.name.prefix` setting! ([#​3666](https://github.com/testcontainers/testcontainers-java/issues/3666)) [@​reda-alaoui](https://github.com/reda-alaoui)\n-   Jackson dependency is now forced to an older version to help with `NoClassDefFoundError (com/fasterxml/jackson/annotation/JsonMerge)`.\n\nAnd more!\n\n##### 🚀 Features & Enhancements\n\n-   Switch to Presto image hosted on GHCR ([#​3667](https://github.com/testcontainers/testcontainers-java/issues/3667)) [@​findepi](https://github.com/findepi)\n-   Implement getDatabaseName() in CockroachContainer ([#​3778](https://github.com/testcontainers/testcontainers-java/issues/3778)) [@​croemmich](https://github.com/croemmich)\n-   Make recorder .flv videos scrollable ([#​512](https://github.com/testcontainers/testcontainers-java/issues/512)) ([#​3180](https://github.com/testcontainers/testcontainers-java/issues/3180)) [@​oussamabadr](https://github.com/oussamabadr)\n-   Support HTTP headers on HttpWaitStrategy ([#​2549](https://github.com/testcontainers/testcontainers-java/issues/2549)) [@​renatomefi](https://github.com/renatomefi)\n-   Show port mappings in HttpWaitStrategy ([#​2341](https://github.com/testcontainers/testcontainers-java/issues/2341)) [@​aguibert](https://github.com/aguibert)\n-   Support newer versions of CockroachDB by changing the docker command ([#​3608](https://github.com/testcontainers/testcontainers-java/issues/3608)) [@​giger85](https://github.com/giger85)\n-   Improve logging for port listener ([#​3736](https://github.com/testcontainers/testcontainers-java/issues/3736)) [@​artjomka](https://github.com/artjomka)\n-   couchbase: wait until all services are part of the config ([#​3003](https://github.com/testcontainers/testcontainers-java/issues/3003)) [@​daschl](https://github.com/daschl)\n-   Support Ryuk socket timeout ([#​3682](https://github.com/testcontainers/testcontainers-java/issues/3682)) [@​diegolovison](https://github.com/diegolovison)\n-   Add init command parameter to Vault container ([#​3188](https://github.com/testcontainers/testcontainers-java/issues/3188)) [@​tandrup](https://github.com/tandrup)\n-   Startables#deepStart with varargs ([#​3261](https://github.com/testcontainers/testcontainers-java/issues/3261)) [@​jochenchrist](https://github.com/jochenchrist)\n\n##### 🐛 Bug Fixes\n\n-   Remove extra newlines in container log output ([#​3752](https://github.com/testcontainers/testcontainers-java/issues/3752)) [@​perlun](https://github.com/perlun)\n-   Fix handling of locally built images when used with `hub.image.name.prefix` ([#​3666](https://github.com/testcontainers/testcontainers-java/issues/3666)) [@​reda-alaoui](https://github.com/reda-alaoui)\n\n##### 📖 Documentation\n\n-   Kafka cluster example ([#​1984](https://github.com/testcontainers/testcontainers-java/issues/1984), [#​3758](https://github.com/testcontainers/testcontainers-java/issues/3758)) [@​seglo](https://github.com/seglo)\n-   Documentation PostGIS JDBC url sample version update ([#​3606](https://github.com/testcontainers/testcontainers-java/issues/3606)) [@​aulea](https://github.com/aulea)\n-   GenericContainer: fix typo in Javadoc ([#​3684](https://github.com/testcontainers/testcontainers-java/issues/3684)) [@​perlun](https://github.com/perlun)\n-   Added documentation for Bigtable Emulator container ([#​3708](https://github.com/testcontainers/testcontainers-java/issues/3708)) [@​RamazanYapparov](https://github.com/RamazanYapparov)\n-   Clarify usage of host port exporting ([#​3421](https://github.com/testcontainers/testcontainers-java/issues/3421)) [@​alxgrk](https://github.com/alxgrk)\n-   Rename Presto to Trino ([#​3649](https://github.com/testcontainers/testcontainers-java/issues/3649)) [@​martint](https://github.com/martint)\n\n##### 🧹 Housekeeping\n\n-   update release drafter to v5.13.0 ([#​3632](https://github.com/testcontainers/testcontainers-java/issues/3632)) [@​jetersen](https://github.com/jetersen)\n-   Remove duplicated dependency in jdbc-test module ([#​3664](https://github.com/testcontainers/testcontainers-java/issues/3664)) [@​giger85](https://github.com/giger85)\n-   add github actions to dependabot config ([#​3633](https://github.com/testcontainers/testcontainers-java/issues/3633)) [@​jetersen](https://github.com/jetersen)\n-   Remove ciMate ([#​3631](https://github.com/testcontainers/testcontainers-java/issues/3631)) [@​bsideup](https://github.com/bsideup)\n\n##### 📦 Dependency updates\n\n-   update release drafter to v5.13.0 ([#​3632](https://github.com/testcontainers/testcontainers-java/issues/3632)) [@​jetersen](https://github.com/jetersen)\n-   Force Jackson version ([#​3602](https://github.com/testcontainers/testcontainers-java/issues/3602)) [@​bsideup](https://github.com/bsideup)\n-   Bump aws-java-sdk-sqs from 1.11.884 to 1.11.930 in /modules/localstack ([#​3660](https://github.com/testcontainers/testcontainers-java/issues/3660)) [@​dependabot](https://github.com/dependabot)\n-   Bump guava from 30.0-jre to 30.1-jre in /modules/jdbc-test ([#​3622](https://github.com/testcontainers/testcontainers-java/issues/3622)) [@​dependabot](https://github.com/dependabot)\n-   Bump aws-java-sdk-s3 from 1.11.929 to 1.11.930 in /modules/localstack ([#​3659](https://github.com/testcontainers/testcontainers-java/issues/3659)) [@​dependabot](https://github.com/dependabot)\n-   Bump org.jetbrains.kotlin.plugin.spring from 1.3.31 to 1.4.21-2 in /examples ([#​3646](https://github.com/testcontainers/testcontainers-java/issues/3646)) [@​dependabot](https://github.com/dependabot)\n-   Bump gradle-update/update-gradle-wrapper-action from [`74a035c`](https://github.com/testcontainers/testcontainers-java/commit/74a035c) to 1.0.9 ([#​3653](https://github.com/testcontainers/testcontainers-java/issues/3653)) [@​dependabot](https://github.com/dependabot)\n-   Bump cucumber-junit from 6.8.1 to 6.9.1 in /examples ([#​3625](https://github.com/testcontainers/testcontainers-java/issues/3625)) [@​dependabot](https://github.com/dependabot)\n-   Bump testcontainers from 1.14.3 to 1.15.1 in /core ([#​3587](https://github.com/testcontainers/testcontainers-java/issues/3587)) [@​dependabot](https://github.com/dependabot)\n-   Bump transport from 7.10.0 to 7.10.1 in /modules/elasticsearch ([#​3590](https://github.com/testcontainers/testcontainers-java/issues/3590)) [@​dependabot](https://github.com/dependabot)\n-   Bump aws-java-sdk-s3 from 1.11.882 to 1.11.929 in /modules/localstack ([#​3654](https://github.com/testcontainers/testcontainers-java/issues/3654)) [@​dependabot](https://github.com/dependabot)\n-   Bump s3 from 2.15.14 to 2.15.56 in /modules/localstack ([#​3650](https://github.com/testcontainers/testcontainers-java/issues/3650)) [@​dependabot](https://github.com/dependabot)\n-   Bump org.jetbrains.kotlin.jvm from 1.3.31 to 1.4.21-2 in /examples ([#​3645](https://github.com/testcontainers/testcontainers-java/issues/3645)) [@​dependabot](https://github.com/dependabot)\n-   Bump guava from 30.0-jre to 30.1-jre in /core ([#​3627](https://github.com/testcontainers/testcontainers-java/issues/3627)) [@​dependabot](https://github.com/dependabot)\n-   Bump ad-m/github-push-action from [`68af989`](https://github.com/testcontainers/testcontainers-java/commit/68af9897f2b021035ca3952bf354bbb4675c1762) to 0.6.0 ([#​3658](https://github.com/testcontainers/testcontainers-java/issues/3658)) [@​dependabot](https://github.com/dependabot)\n-   Bump aws-java-sdk-dynamodb from 1.11.901 to 1.11.929 in /modules/dynalite ([#​3652](https://github.com/testcontainers/testcontainers-java/issues/3652)) [@​dependabot](https://github.com/dependabot)\n-   Bump gradle/wrapper-validation-action from [`e2c57ac`](https://github.com/testcontainers/testcontainers-java/commit/e2c57ac) to 1.0.3 ([#​3657](https://github.com/testcontainers/testcontainers-java/issues/3657)) [@​dependabot](https://github.com/dependabot)\n-   Update actions/cache requirement to v2.1.3 ([#​3651](https://github.com/testcontainers/testcontainers-java/issues/3651)) [@​dependabot](https://github.com/dependabot)\n-   Bump solr-solrj from 8.6.3 to 8.7.0 in /examples ([#​3446](https://github.com/testcontainers/testcontainers-java/issues/3446)) [@​dependabot](https://github.com/dependabot)\n-   Bump cucumber-java from 6.8.1 to 6.9.1 in /examples ([#​3624](https://github.com/testcontainers/testcontainers-java/issues/3624)) [@​dependabot](https://github.com/dependabot)\n-   Bump r2dbc-mariadb from 0.8.4-rc to 1.0.0 in /modules/mariadb ([#​3593](https://github.com/testcontainers/testcontainers-java/issues/3593)) [@​dependabot](https://github.com/dependabot)\n-   Bump org.springframework.boot from 2.3.4.RELEASE to 2.4.1 in /examples ([#​3594](https://github.com/testcontainers/testcontainers-java/issues/3594)) [@​dependabot](https://github.com/dependabot)\n-   Bump jedis from 3.3.0 to 3.4.0 in /examples ([#​3595](https://github.com/testcontainers/testcontainers-java/issues/3595)) [@​dependabot](https://github.com/dependabot)\n-   Bump tomcat-jdbc from 9.0.40 to 10.0.0 in /modules/jdbc ([#​3592](https://github.com/testcontainers/testcontainers-java/issues/3592)) [@​dependabot](https://github.com/dependabot)\n-   Bump elasticsearch-rest-client from 7.10.0 to 7.10.1 in /modules/elasticsearch ([#​3589](https://github.com/testcontainers/testcontainers-java/issues/3589)) [@​dependabot](https://github.com/dependabot)\n-   Bump jedis from 3.3.0 to 3.4.0 in /core ([#​3586](https://github.com/testcontainers/testcontainers-java/issues/3586)) [@​dependabot](https://github.com/dependabot)\n-   Bump rest-assured from 4.3.1 to 4.3.3 in /modules/vault ([#​3588](https://github.com/testcontainers/testcontainers-java/issues/3588)) [@​dependabot](https://github.com/dependabot)\n-   Bump r2dbc-mssql from 0.8.4.RELEASE to 0.8.5.RELEASE in /modules/mssqlserver ([#​3397](https://github.com/testcontainers/testcontainers-java/issues/3397)) [@​dependabot](https://github.com/dependabot)\n-   Bump mockito-core from 3.5.15 to 3.6.28 in /modules/junit-jupiter ([#​3539](https://github.com/testcontainers/testcontainers-java/issues/3539)) [@​dependabot](https://github.com/dependabot)\n-   Bump mockito-core from 3.5.13 to 3.6.28 in /core ([#​3542](https://github.com/testcontainers/testcontainers-java/issues/3542)) [@​dependabot](https://github.com/dependabot)\n-   Bump mssql-jdbc from 9.1.0.jre8-preview to 9.1.1.jre8-preview in /modules/mssqlserver ([#​3563](https://github.com/testcontainers/testcontainers-java/issues/3563)) [@​dependabot](https://github.com/dependabot)\n-   Bump pulsar-client-admin from 2.6.1 to 2.7.0 in /modules/pulsar ([#​3566](https://github.com/testcontainers/testcontainers-java/issues/3566)) [@​dependabot](https://github.com/dependabot)\n-   Bump pulsar-client from 2.6.1 to 2.7.0 in /modules/pulsar ([#​3565](https://github.com/testcontainers/testcontainers-java/issues/3565)) [@​dependabot](https://github.com/dependabot)\n-   Bump json from [`2018081`](https://github.com/testcontainers/testcontainers-java/commit/20180813) to [`2020111`](https://github.com/testcontainers/testcontainers-java/commit/20201115) in /examples ([#​3567](https://github.com/testcontainers/testcontainers-java/issues/3567)) [@​dependabot](https://github.com/dependabot)\n-   Bump influxdb-java from 2.20 to 2.21 in /modules/influxdb ([#​3568](https://github.com/testcontainers/testcontainers-java/issues/3568)) [@​dependabot](https://github.com/dependabot)\n-   Bump assertj-core from 3.18.0 to 3.18.1 in /modules/vault ([#​3486](https://github.com/testcontainers/testcontainers-java/issues/3486)) [@​dependabot](https://github.com/dependabot)\n-   Upgrade Ryuk to 0.3.1 ([#​3629](https://github.com/testcontainers/testcontainers-java/issues/3629)) [@​rnorth](https://github.com/rnorth)\n\n\n\n---\n\n### Renovate configuration\n\n:date: **Schedule**: At any time (no schedule defined).\n\n:vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied.\n\n:recycle: **Rebasing**: Whenever PR is behind base branch, or rename PR to start with \"rebase!\".\n\n:no_bell: **Ignore**: Close this PR and you won't be reminded about these updates again.\n\n---\n\nThis PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).",
           "fromRef": {"id": "refs/heads/renovate/org.testcontainers.version"},
           "toRef": {"id": "refs/heads/develop"},
           "reviewers": []
         },
         "responseType": "json",
         "headers": {
           "accept-encoding": "gzip, deflate",
           "X-Atlassian-Token": "no-check",
           "user-agent": "https://github.com/renovatebot/renovate",
           "authorization": "***********"
         },
         "hooks": {"beforeRedirect": [null]},
         "timeout": 60000
       }

Conclusion

I assume this happens, because the release-notes contain special characters, which are not "escaped"/fixed correctly.

I expected that - when the unicodeEmoji is set to false - it should escape/fix the emojis or at least remove them.

Possible workarounds or solutions

• Open the PR manually (we do that for that dependency at the moment)
• Modify https://docs.renovatebot.com/configuration-options/#prbodytemplate so that it excludes changelogs
• If Bitbucket/Atlassian would fix this issue we could modify the collation of MySQL so that it uses utf8mb4 → https://stackoverflow.com/a/35189650

[rarkins]

This is the function which emojifies or not: https://github.com/renovatebot/renovate/blob/19e16ff7f0feb9228a8e40de20d67fd2c690ee70/lib/util/emoji.ts#L10-L12

But maybe the problem is that the emojies are already in the changelog. In which case we need that function to try to de-emoji too?

[AB-xdev]

@rarkins

But maybe the problem is that the emojies are already in the changelog.

Yes, I think this is exactly my problem.

[rarkins]

Looks like emoji.strip or emoji.replace might work, but it would need testing.

[renovate-release]

:tada: This issue has been resolved in version 24.115.0 :tada:

The release is available on:

• GitHub release
• 24.115.0

Your semantic-release bot :package::rocket:

[AB-xdev]

@rarkins Thank you for the fix. Unfortunately I wasn't able to test it until now.

Emojis like 🚀 are now filtered 👍

However the fix only seems to only work partially. The pullrequest creation still fails, because there is somehow a "broom" emoji 🧹 included (which should have been filtered out):

full log-file: renovate.txt

[viceice]

@AB-xdev Please open a new bug issue with full details. Rememer you need to set unicodeEmoji=false to get it working.