Signature validation failed on some emojis

[ddhp]

Bug Report

Describe the bug A similar issue had been filed in the php sdk repo a few years ago.

We tried the solution concluded from the issue. Basically we can see LINE message API would escape emoji to unicode string inside the message body, e.g 🤨 -> \uD83E\uDD28 before calculating signature, so we also modify the message body the same way.

But this solution doesn't work for all the emojis. AFAIK, ✋✊ won't be escaped by LINE message api.

So in order for our apps to validate signatures properly, can LINE provide us the rule of escaping emoji and the range of unicodes would be escaped?

Thank you

To Reproduce follow signature validation from the official guide and send some emojis to the OA.

Expected behavior Able to validate signature from message api webhook

Screenshots N/A

Environment (please complete the following information): N/A

Additional context N/A

[eucyt]

Thank you for your question. Firstly, for signature verification, it is necessary to use the body as it is. Therefore, the webhook recipient don't have to rewrite body.

However, It is possible that Twilio Serverless or JSON.stringify is unintentionally changing characters, and you might need to specifically escape emojis to revert this.

Please ensure that the received webhook, including spaces, has not been altered.

[ddhp]

Hi @eucyt, thanks for your reply!

We didn't use Twilio Serverless, we directly use JSON.stringify to parse JSON payload into string. I've looked up and JSON.stringify wouldn't escape emojis, e.g JSON.stringify({a: '🤨'}) // '{"a":"🤨"}'

Code below is from LINE dev documents, and if we tried body as {text: JSON.stringify("🤨")}, the calculated signature wouldn't match the one presented in request header

const crypto = require("crypto");

const channelSecret = "..."; // Channel secret string
const body = JSON.stringify("{ text: 🤨}"); // Request body string
const signature = crypto
  .createHmac("SHA256", channelSecret)
  .update(body)
  .digest("base64");
// Compare x-line-signature request header and the signature

but if we escaped the emojis, i.e make the body as JSON.stringify({text: "\uD83E\uDD28"}), it matches the signature in request header. (message api received 🤨 and signature is calculated as \uD83E\uDD28)

Does this mean LINE message API would escape some unicodes to string? Or are we missing something else?

Thanks for your reply in advance!

[eucyt]

It is likely that this issue is caused not by the escaping of emojis, but by JSON.stringify removing spaces or changing some charactors. JSON.stringify converts {a: '🤨'} to {"a":"🤨"} (with spaces removed and so on). This diff from the request body cause a signature verification failure.

Could you please verify the signature without using JSON.stringify, as described in this document?

const crypto = require("crypto");

const channelSecret = "..."; // Channel secret string
const body = "..."; // Request body string
const signature = crypto
  .createHmac("SHA256", channelSecret)
  .update(body)
  .digest("base64");
// Compare x-line-signature request header and the signature

[ddhp]

in the example, .update only takes type string or NodeJS.ArrayBufferView, so we have to make JSON body into a string by JSON.stringify

code below is with a real payload body from message API webhook

const receivedBody = {"destination":"uce28b96d590a32aca613ed9401d290fa","events":[{"type":"message","message":{"type":"text","id":"521148777382543885","quoteToken":"qj_8_gD5-JSjuOiDF4tBP0ijL2m0jMUNNXbKIcKY2P7fTLAE0qMAUhuP0z8UiWh5OKZMlB5g5uPUou8lD9Wbotsb60KfiNO6ZHSzIYIAb2BQzKsmK8ZQPM0zjuBZ2ystszRajBVXGqRyCWi6pKlUSg","text":"🤨"},"webhookEventId":"01J533PQSZMT61R0H1W2X3Q73Z","deliveryContext":{"isRedelivery":false},"timestamp":1723460181639,"source":{"type":"user","userId":"u6afc4dac41d4a41969852de5d4b28a0c"},"replyToken":"53442902dafb45f18a7962ecb1b2daab","mode":"active"}]}
const bodyInString = JSON.stringify(receivedBody)
// '{"destination":"uce28b96d590a32aca613ed9401d290fa","events":[{"type":"message","message":{"type":"text","id":"521148777382543885","quoteToken":"qj_8_gD5-JSjuOiDF4tBP0ijL2m0jMUNNXbKIcKY2P7fTLAE0qMAUhuP0z8UiWh5OKZMlB5g5uPUou8lD9Wbotsb60KfiNO6ZHSzIYIAb2BQzKsmK8ZQPM0zjuBZ2ystszRajBVXGqRyCWi6pKlUSg","text":"🤨"},"webhookEventId":"01J533PQSZMT61R0H1W2X3Q73Z","deliveryContext":{"isRedelivery":false},"timestamp":1723460181639,"source":{"type":"user","userId":"u6afc4dac41d4a41969852de5d4b28a0c"},"replyToken":"53442902dafb45f18a7962ecb1b2daab","mode":"active"}]}'

original JSON body doesn't have any space between key and value and there is no space after JSON.stringify as well. AFAICS JSON.stringify is doing as expected.

Is there anything else we should check?

Have you try sending that emoji to any OA? We tried it on other OA and we can reproduce it as well(no reply from OA)

Thank you

[eucyt]

I'm sorry for the delayed response and any confusion caused.

Let's organize our thoughts.

Why is it necessary to verify the signature with a Webhook?

The main reasons are to verify the sender and to ensure that the body has not been tampered with. In other words, you should verify the raw body received.

The root cause of the current error

The root cause of the current error is that the received body has already been deserialized into a dictionary, and you are trying to re-serialize it into a string by JSON.stringify. This cannot be considered a raw body and is not suitable for verification.

For example,🤨 and ✋✊ were correctly verified on a simple Node server like the one below. This indicates that the type of emoji is not the root cause of the verification failure.

const http = require("http");
const crypto = require("crypto");

const server = http.createServer((req, res) => {
  const channelSecret = "..."; // Channel secret string
  let body = "";
  req.on("data", (chunk) => {
    body += chunk.toString();
  });

  req.on("end", () => {
    // Signature verification
    const signature = crypto
      .createHmac("SHA256", channelSecret)
      .update(body)
      .digest("base64");
    console.log(signature == req.headers["x-line-signature"]);

    res.statusCode = 204;
    res.end();
  });
});

const port = 3000;
const hostname = "127.0.0.1";

server.listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:${port}/`);
});

Solution

To reiterate, the solution is to use the raw body as it is for verification. I am not sure how to do this with Twilio, but if you cannot obtain the raw request body, you will not be able to verify any webhook, not just LINE. We recommend using frameworks that can correctly obtain the raw body, such as Express, Hono, or a simple Node server.

Thank you for using LINE's Webhook. I hope your issue gets resolved.

[ddhp]

Thanks for your reply! We solved it!

The problem is from express.json which is a middleware parse payload body to JSON.

When user send 🤨, LINE message API would send escaped string \uD83E\uDD28 as body to webhook endpoint. After it's processed by express.json, it will be encoded to 🤨, so signature calculated from the json body after JSON.stringify wouldn't match.

Use express.raw to parse payload body will solve this.

Thanks again