Replies to some formatted Matrix messages are not forwarded to IRC

brycied00d commented 5 years ago

It appears that replying to some Matrix messages will fail to relay the message to IRC and throws the following exception:

2018-11-01 18:21:09 INFO:req [d196lp4bs7400] [M->I] m.room.message usr=@piroko:pintobyte.com rm=!pKgHLFCEORYdDgndcZ:matrix.org body=> <@lteo:pintobyte.c
2018-11-01 18:21:09 INFO:req [d196lp4bs7400] [M->I] Relaying message in #devious on irc.freenode.net
2018-11-01 18:21:09 ERROR:main TypeError: Cannot read property 'length' of null
    at MatrixHandler.<anonymous> (/home/matrix-synapse/matrix-appservice-irc/lib/bridge/MatrixHandler.js:1529:15)
    at Generator.next (<anonymous>)
    at Generator.tryCatcher (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/util.js:16:23)
    at PromiseSpawn._promiseFulfilled (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/generators.js:97:49)
    at MatrixHandler._textForReplyEvent (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/generators.js:201:15)
    at MatrixHandler.<anonymous> (/home/matrix-synapse/matrix-appservice-irc/lib/bridge/MatrixHandler.js:1282:34)
    at Generator.next (<anonymous>)
    at Generator.tryCatcher (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/util.js:16:23)
    at PromiseSpawn._promiseFulfilled (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/generators.js:97:49)
    at MatrixHandler._sendIrcAction (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/generators.js:201:15)
    at /home/matrix-synapse/matrix-appservice-irc/lib/bridge/MatrixHandler.js:1246:28
    at Generator.next (<anonymous>)
    at Generator.tryCatcher (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/util.js:16:23)
    at PromiseSpawn._promiseFulfilled (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/generators.js:97:49)
    at Async._drainQueue (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/async.js:138:12)
    at Async._drainQueues (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:810:20)
    at tryOnImmediate (timers.js:768:5)
    at processImmediate [as _immediateCallback] (timers.js:745:5)
2018-11-01 18:21:09 INFO:IrcBridge [d196lp4bs7400] [M->I] FAILED  (3ms)

I was able to produce the issue when replying to the following Matrix messages:

{
  "origin_server_ts": 1541096469463,
  "sender": "@piroko:pintobyte.com",
  "event_id": "$154109646917ITWPn:pintobyte.com",
  "unsigned": {
    "age": 192
  },
  "content": {
    "body": "> <@lteo:pintobyte.com> \"With the launch of the iPhone XS, iPhone XS Max, and iPhone XR in September 2018, Apple discontinued the iPhone SE and stopped selling the device\"\n> \n> oh eff\n\nyeah :( That's mostly what pushed me to getting the XS",
    "m.relates_to": {
      "m.in_reply_to": {
        "event_id": "$154104184912198vlDBy:pintobyte.com"
      }
    },
    "msgtype": "m.text",
    "formatted_body": "<mx-reply><blockquote><a href=\"https://matrix.to/#/!pKgHLFCEORYdDgndcZ:matrix.org/$154104184912198vlDBy:pintobyte.com\">In reply to</a> <a href=\"https://matrix.to/#/@lteo:pintobyte.com\">@lteo:pintobyte.com</a><br>\"With the launch of the iPhone XS, iPhone XS Max, and iPhone XR in September 2018, Apple discontinued the iPhone SE and stopped selling the device\"\n\noh eff</blockquote></mx-reply>yeah :( That's mostly what pushed me to getting the XS",
    "format": "org.matrix.custom.html"
  },
  "type": "m.room.message",
  "room_id": "!pKgHLFCEORYdDgndcZ:matrix.org"
}

For reference, the above was in reply to this message:

{
  "origin_server_ts": 1541041849825,
  "sender": "@lteo:pintobyte.com",
  "event_id": "$154104184912198vlDBy:pintobyte.com",
  "unsigned": {
    "age": 404
  },
  "content": {
    "body": "\"With the launch of the iPhone XS, iPhone XS Max, and iPhone XR in September 2018, Apple discontinued the iPhone SE and stopped selling the device\"\n\noh eff",
    "msgtype": "m.text"
  },
  "type": "m.room.message",
  "room_id": "!pKgHLFCEORYdDgndcZ:matrix.org"
}

And another failed reply:

{
  "origin_server_ts": 1541097004350,
  "sender": "@brycec:cobryce.com",
  "event_id": "$154109700460cwKCN:cobryce.com",
  "unsigned": {
    "age": 2751,
    "transaction_id": "m1541097003311.23"
  },
  "content": {
    "body": "> <@brycec:cobryce.com> Maybe try not throwing exceptions, mmkay? 😛 \n> ```\n> 2018-11-01 18:21:09 INFO:req [d196lp4bs7400] [M->I] m.room.message usr=@piroko:pintobyte.com rm=!pKgHLFCEORYdDgndcZ:matrix.org body=> <@lteo:pintobyte.c\r\n> 2018-11-01 18:21:09 INFO:req [d196lp4bs7400] [M->I] Relaying message in #devious on irc.freenode.net\r\n> 2018-11-01 18:21:09 ERROR:main TypeError: Cannot read property 'length' of null                                                                                                                                                                                                         \r\n>     at MatrixHandler.<anonymous> (/home/matrix-synapse/matrix-appservice-irc/lib/bridge/MatrixHandler.js:1529:15)                                                                                                                                              \r\n>     at Generator.next (<anonymous>)                                                                \r\n>     at Generator.tryCatcher (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/util.js:16:23)\r\n>     at PromiseSpawn._promiseFulfilled (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/generators.js:97:49)                                                                                                                                               \r\n>     at MatrixHandler._textForReplyEvent (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/generators.js:201:15)                                                                                                                   \r\n>     at MatrixHandler.<anonymous> (/home/matrix-synapse/matrix-appservice-irc/lib/bridge/MatrixHandler.js:1282:34)                                                \r\n>     at Generator.next (<anonymous>)                                                                                               \r\n>     at Generator.tryCatcher (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/util.js:16:23)                                                    \r\n>     at PromiseSpawn._promiseFulfilled (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/generators.js:97:49)                                    \r\n>     at MatrixHandler._sendIrcAction (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/generators.js:201:15)                         \r\n>     at /home/matrix-synapse/matrix-appservice-irc/lib/bridge/MatrixHandler.js:1246:28                                                                   \r\n>     at Generator.next (<anonymous>)                                                                 \r\n>     at Generator.tryCatcher (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/util.js:16:23)                                                    \r\n>     at PromiseSpawn._promiseFulfilled (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/generators.js:97:49)\r\n>     at Async._drainQueue (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/async.js:138:12)                                \r\n>     at Async._drainQueues (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/async.js:143:10)                                                                                                  \r\n>     at Immediate.Async.drainQueues (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/async.js:17:14)\r\n>     at runCallback (timers.js:810:20)                                                           \r\n>     at tryOnImmediate (timers.js:768:5)                                                                                                                                                                                                                                                 \r\n>     at processImmediate [as _immediateCallback] (timers.js:745:5)                                                                                                                                                                                              \r\n> 2018-11-01 18:21:09 INFO:IrcBridge [d196lp4bs7400] [M->I] FAILED  (3ms)                  \r\n> ```\n\nWell aint that weird... I ALSO \"cannot reply\" (throws the same exception) to that message. But another message from yesterday was okay. I have no idea what's up. :/",
    "format": "org.matrix.custom.html",
    "msgtype": "m.text",
    "formatted_body": "<mx-reply><blockquote><a href=\"https://matrix.to/#/!pKgHLFCEORYdDgndcZ:matrix.org/$154109684155xmJyF:cobryce.com\">In reply to</a> <a href=\"https://matrix.to/#/@brycec:cobryce.com\">@brycec:cobryce.com</a><br><p>Maybe try not throwing exceptions, mmkay? 😛</p>\n<pre><code>2018-11-01 18:21:09 INFO:req [d196lp4bs7400] [M-&gt;I] m.room.message usr=@piroko:pintobyte.com rm=!pKgHLFCEORYdDgndcZ:matrix.org body=&gt; &lt;@lteo:pintobyte.c\n2018-11-01 18:21:09 INFO:req [d196lp4bs7400] [M-&gt;I] Relaying message in #devious on irc.freenode.net\n2018-11-01 18:21:09 ERROR:main TypeError: Cannot read property 'length' of null                                                                                                                                                                                                         \n    at MatrixHandler.&lt;anonymous&gt; (/home/matrix-synapse/matrix-appservice-irc/lib/bridge/MatrixHandler.js:1529:15)                                                                                                                                              \n    at Generator.next (&lt;anonymous&gt;)                                                                \n    at Generator.tryCatcher (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/util.js:16:23)\n    at PromiseSpawn._promiseFulfilled (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/generators.js:97:49)                                                                                                                                               \n    at MatrixHandler._textForReplyEvent (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/generators.js:201:15)                                                                                                                   \n    at MatrixHandler.&lt;anonymous&gt; (/home/matrix-synapse/matrix-appservice-irc/lib/bridge/MatrixHandler.js:1282:34)                                                \n    at Generator.next (&lt;anonymous&gt;)                                                                                               \n    at Generator.tryCatcher (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/util.js:16:23)                                                    \n    at PromiseSpawn._promiseFulfilled (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/generators.js:97:49)                                    \n    at MatrixHandler._sendIrcAction (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/generators.js:201:15)                         \n    at /home/matrix-synapse/matrix-appservice-irc/lib/bridge/MatrixHandler.js:1246:28                                                                   \n    at Generator.next (&lt;anonymous&gt;)                                                                 \n    at Generator.tryCatcher (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/util.js:16:23)                                                    \n    at PromiseSpawn._promiseFulfilled (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/generators.js:97:49)\n    at Async._drainQueue (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/async.js:138:12)                                \n    at Async._drainQueues (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/async.js:143:10)                                                                                                  \n    at Immediate.Async.drainQueues (/home/matrix-synapse/matrix-appservice-irc/node_modules/bluebird/js/release/async.js:17:14)\n    at runCallback (timers.js:810:20)                                                           \n    at tryOnImmediate (timers.js:768:5)                                                                                                                                                                                                                                                 \n    at processImmediate [as _immediateCallback] (timers.js:745:5)                                                                                                                                                                                              \n2018-11-01 18:21:09 INFO:IrcBridge [d196lp4bs7400] [M-&gt;I] FAILED  (3ms)                  \n</code></pre>\n</blockquote></mx-reply>Well aint that weird... I ALSO &quot;cannot reply&quot; (throws the same exception) to that message. But another message from yesterday was okay. I have no idea what's up. :/",
    "m.relates_to": {
      "m.in_reply_to": {
        "event_id": "$154109684155xmJyF:cobryce.com"
      }
    }
  },
  "type": "m.room.message",
  "room_id": "!pKgHLFCEORYdDgndcZ:matrix.org"
}

Versions

Observed while running: matrix-synapse 0.33.8 matrix-appservice-irc tag 0.11.2 (26f39c9fea97a9c94421ee02d8d02f1ed66914f8) bridged to Freenode

Steps to reproduce

As the issue appears to be caused by replying to a message that begins with some form of Markdown-based formatting, specifically block-quote or ``` quoting, then to reproduce:

Send (or have someone send) a message that begins with a quote.
Reply to that message.
Observe that the reply is never brdiged, and a backtrace is produced on the IRC bridge console/log.

Cause

Based on the backtrace, it appears to be caused by a regular expression failing to match the message body, and there being no graceful handling of such a situation. Specifically, the regular expression expects two consecutive newlines, however the markdown-to-plaintext routine prefixes the second newline with the raw markdown. Perhaps the regular expression should not expect that second consecutive newline, or accept one/some wildcard matches before that second newline. https://github.com/matrix-org/matrix-appservice-irc/blob/master/lib/bridge/MatrixHandler.js#L1524-L1531

TildeSlashC0re commented 5 years ago

just dropping by to confirm this!

I am also running into issues with replies not being bridged, rather randomly.

TildeSlashC0re commented 5 years ago

spec:

To strip the fallback on the body, the client should iterate over each line of the string, removing any lines that start with the fallback prefix (">", including the space, without quotes) and stopping when a line is encountered without the prefix. This prefix is known as the "fallback prefix sequence".

irc bridge:

match /> <(@.*:.*)>(.*)\n\n(.*)/ in event body and use the third capture group as the fallback-free body

-@tulir

anyway the fix is to remove reply fallbacks properly in the irc bridge (i.e. remove all lines starting with > until a blank line is reached) that can be done without regex by splitting it by newline, removing the fallback lines and joining by newline

-@tulir

TildeSlashC0re commented 5 years ago

adding that this is tangentially related to #683 (measured by the result == loss of message)

Mikaela commented 5 years ago

Is this the same issue?

Screenshot from 2019-07-27 22-52-24

Screenshot from 2019-07-27 22-52-54

matrix-org / matrix-appservice-irc