Issue with Line Breaks (multiline)

HongDuHyeon commented 1 year ago

Orb version:

circleci/slack@4.12.5

What happened:

Hello, I'm using CircleCI to automate repetitive tasks from GitHub. However, I've encountered an error that is causing delays. The issue is related to sending the body of a master pull request on the GitHub master branch to a specific channel on our team's Slack using the provided format:

  github-master-pr:
    docker:
      - image: circleci/python:3.8
    steps:
      - checkout

      - run:
          name: Send PR details to Slack
          command: |
              export PR_NUMBER=$(echo "${CIRCLE_PULL_REQUEST}" | awk -F'/' '{print $NF}')
              export PR_RESPONSE=$(curl -s -H "Authorization: Bearer <github-access-token>" "https://api.github.com/repos/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/pulls/${PR_NUMBER}")

              if [[ "$PR_RESPONSE" == *"message"* ]]; then
                echo "Failed to fetch PR details: $PR_RESPONSE"
                exit 1
              fi

              export PR_BODY=$(echo -e "$PR_RESPONSE" | jq -r '.body')

              export PR_MESSAGE="$PR_BODY"

              echo "export MESSAGE=${PR_MESSAGE}" >> $BASH_ENV
      - slack/notify:
          channel: "notice"
          custom: |
            {
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "master merge pull requests \n ${MESSAGE}"
                  }
                }
              ]
            }

It's not completely failing to send the message. As part of the master pull request, I include information about merged pull requests from the develop branch in the body. Here's an example:

ex). (#123) first merge pull request (#124) second merge pull request (#125) third merge pull request ...

I have successfully fetched the data from GitHub, and it is being displayed. However, the handling of line breaks \n is not working properly, which requires me to write everything in a single line. Additionally, I'm encountering the following error in CircleCI:

"parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 7, column 232"

Expected behavior:

Firstly, I attempted to add escaping in my code with the following line:

export PR_BODY=$(echo "$PR_RESPONSE" | jq -r '.body' | sed -z 's/\n/\\n/g')

However, it didn't work, and I confirmed that the \n characters still appeared as is.

Secondly, I checked the official CircleCI documentation, and there's a suspicious part that caught my attention: https://github.com/CircleCI-Public/slack-orb/blob/main/src/scripts/notify.sh#L17 Line: 17

CUSTOM_BODY_MODIFIED=$(echo "$CUSTOM_BODY_MODIFIED" | sed 's/\\/\\\\/g' | sed 's/"/\\"/g' | sed 's/`/\\`/g')

add : I was considering adding code to escape \n in this section to see if it resolves the issue.

CUSTOM_BODY_MODIFIED=$(echo "$CUSTOM_BODY_MODIFIED" | sed 's/\\/\\\\/g' | sed 's/"/\\"/g' | sed 's/`/\\`/g' | sed 's/\\n/\\\\n/g'))")

However, if there's already a known solution or answer regarding this problem, I would greatly appreciate it.

Additional Information:

The current output appears as follows:

master merge pull requests 
(#123) first merge pull request\r\n(#124) second merge pull request\r\n(#125) third merge pull request

reynoldsme commented 12 months ago

Also seeing this behavior in a previously working configuration after bumping circleci/slack from 4.6.1 to 4.12.5.

0A line feed characters stored in a shell variable and interpolated in a custom template like so "text": "*Message*: $GIT_COMMIT_DESC" are rendered as the string "\n" in slack messages.

HongDuHyeon commented 12 months ago

Are you experiencing the same issue as me?

reynoldsme commented 12 months ago

Yes, looks like it is the same bug. The difference is your example appears to contain CRLF (\r\n) Windows style line endings while mine are all Unix style LF (\n) line endings. I'm just establishing that the bug is not specific to a particular type of of line ending.

fszymanski-blvd commented 11 months ago

I managed to get something somewhat working based on another user's comment on a different thread by piping to tr -d '\r'. Since I was loading the data from a github url, the redacted command looked like echo "export RELEASE_TAG_BODY=\"\$(curl -L -H \"Accept: application/vnd.github+json\" -H \"Authorization: Bearer $TOKEN_ENVVAR\" -H \"X-GitHub-Api-Version: 2022-11-28\" https://api.github.com/repos/Org/repo/releases/tags/$tag | jq -r .body | tr -d '\r')\"" >> "$BASH_ENV"

Ofer-CO commented 10 months ago

getting the same behavior, seems to come from the sanitizing process that is added in the last version

EricRibeiro commented 8 months ago

Hey folks 👋

Thank you for bringing this to our notice, and sorry for not getting to it sooner. Special characters have been a recurring issue with this orb, and we have tried, without much success (😅), to escape them manually using Bash.

We are trying a different approach now. We have a proof-of-concept Slack orb wholly written in Go, which means that we can leverage built-in libraries to do the special characters escaping inside the JSON templates for us.

Would you all be open to trying it and seeing if it solves your issue? We are looking for some early feedback. All you have to do is replace the orb version 4.12.5 in your config with dev:4463bcaee946c1e91829fdf6b3e0d1a7b6f12210:

version: 2.1

orbs:
-  slack: circleci/slack@4.12.5
+  slack: circleci/slack@dev:4463bcaee946c1e91829fdf6b3e0d1a7b6f12210

Please remember that dev:4463bcaee946c1e91829fdf6b3e0d1a7b6f12210 is a proof-of-concept, so it doesn't have feature parity with v4.12.5. As of the time of this writing, it doesn't support sub-shells expansions inside the templates:

{
    "blocks": [
        {
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": "$(cat /tmp/msg)"
            }
        }
    ]
}

Thanks in advance, and let us know if it works for you! 🙇

Warning dev:4463bcaee946c1e91829fdf6b3e0d1a7b6f12210 will be deleted after 90 days (January 22, 2024). Remember to roll back to a production version before the developer version gets deleted.

HongDuHyeon commented 7 months ago

Hello, Eric Ribeiro. Thank you for your response. In that case, I will proceed as you have instructed. Am I correct in understanding that the only method currently available is to develop using the information you provided and that, after the deletion on January 22, 2024, the only option will be to roll back the code? 🥲

EricRibeiro commented 7 months ago

Hello, Eric Ribeiro. Thank you for your response. In that case, I will proceed as you have instructed. Am I correct in understanding that the only method currently available is to develop using the information you provided and that, after the deletion on January 22, 2024, the only option will be to roll back the code? 🥲

We plan on releasing a production-ready version of the rewritten orb soon. I can share another if we don't release it by the time this developer version expires.

Additionally, we are still seeking feedback, so please let us know how it works for you 🙇‍♂️

HongDuHyeon commented 7 months ago

Hello, Eric Ribeiro. Thank you for your response. In that case, I will proceed as you have instructed. Am I correct in understanding that the only method currently available is to develop using the information you provided and that, after the deletion on January 22, 2024, the only option will be to roll back the code? 🥲

We plan on releasing a production-ready version of the rewritten orb soon. I can share another if we don't release it by the time this developer version expires.

Additionally, we are still seeking feedback, so please let us know how it works for you 🙇‍♂️

Thank you very much for your consideration. If there's an opportunity to use the developer version, I would be eager to try it out and provide feedback.🙆‍♂️ Also, if it's not too much trouble, could you kindly inform me here about any new releases? Your assistance would be deeply appreciated. 😭

EricRibeiro commented 7 months ago

Will do! For now, I would appreciate it if you tried the version I shared above.

We have a newer release that you can try if you are feeling adventurous, but it contains breaking changes. Most noticeable is the exclusion of the custom parameters and the inclusion of several other template parameters. This is likely what the next major version of the Slack Orb will follow, so your feedback would be much appreciated.

HongDuHyeon commented 7 months ago

version: 2.1

orbs:
  slack: circleci/slack@dev:4463bcaee946c1e91829fdf6b3e0d1a7b6f12210

  github-master-pr:
    docker:
      - image: circleci/python:3.8
    steps:
      - checkout
      - run:
          name: Send PR details to Slack
          command: |
              export PR_NUMBER=$(echo "${CIRCLE_PULL_REQUEST}" | awk -F'/' '{print $NF}')
              export PR_RESPONSE=$(curl -s -H "Authorization: Bearer [MY_GITHUB_TOKKEN]" "https://api.github.com/repos/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/pulls/${PR_NUMBER}")

              # log
              echo "PR_TITLE: ${PR_TITLE}"
              echo "PR_BODY: ${PR_BODY}"

              if [[ "$PR_RESPONSE" == *"message"* ]]; then
                echo "Failed to fetch PR details: $PR_RESPONSE"
                exit 1
              fi

              export PR_TITLE=$(echo "$PR_RESPONSE" | jq -r '.title')
              export PR_BODY=$(echo "$PR_RESPONSE" | jq -r '.body')

              export MESSAGE="(#${PR_NUMBER})${PR_TITLE}${PR_BODY}"
              echo "MESSAGE: ${MESSAGE}"
              echo "export MESSAGE='${MESSAGE}'" >> $BASH_ENV
      - slack/notify:
          custom: |
            {
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "plain_text",
                    "text": "slack message ${MESSAGE}",
                    "emoji": true
                  }
                }
              ]
            }

When I write it as above, I don't receive any alerts. Could there be something I've overlooked? There are no error messages either.

++ I have successfully passed all six cases: Spin up environment, Preparing environment variables, Checkout code, Send PR details to Slack, Slack - Detecting Job Status (PASS), Slack - Sending Notification, and there were no visible errors.

Even when I send a general message instead of specific data in the text field, I still don't receive an alert.

- slack/notify:
          custom: |
            {
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "plain_text",
                    "text": "slack message",
                    "emoji": true
                  }
                }
              ]
            }

Could there be something I've missed or overlooked? I would appreciate your help.

++ log is...

PR_TITLE: 
PR_BODY: 
MESSAGE: (#4664)Circle CI Slack test1
Circle CI Slack test2
Circle CI Slack test3

However, even though the logs appear normal, I still don't receive notifications on Slack. I'm starting to wonder if it's just a version issue. Notifications worked properly in versions 4.12.5 and 4.6.1.

CircleCI-Public / slack-orb