webhook hangs at webserver "Received an unrecognised webhook"

[horstepipe]

Hello hope I'm not overseeing something. I setup the gateway without docker. After starting it, I'm getting a warning - which might be the problem - but I'm not sure:

2021-10-28 20:17:48.577.000 WARN [bridge] [-] POST http://127.0.0.1:8008/_matrix/client/r0/register (AS) HTTP 400 Error: "{\"errcode\":\"M_USER_IN_USE\",\"error\":\"User ID already taken.\"}"

I also took a look at the homeserver.log, unfortunately there's just the same error. So I'm unsure whether this is critical.

Webhook creation worked fine in Matrix, but if I want to send some test messages, e.g.

 curl -X POST 127.0.0.1:8020/hook/xxxxx -H "Content-Type: application/json" -d '{"Test":"test"}'

I'm seeing this output and the hook is not being passed to the room:

 2021-10-28 20:25:20.570.000      WARN   [webhook-srv]           Received an unrecognised webhook:
{
  Test: 'test'
}

Would be nice if you have some idea about that. Thanks!

[horstepipe]

worth to mention that I'm using Synapse's LDAP Auth provider https://github.com/matrix-org/matrix-synapse-ldap3 and disabled the local user database in homerserver.yaml.

[geluk]

Hi @horstepipe, I do know that the warning you're seeing is unlikely to be the cause of your issue. This warning unfortunately always shows when the application starts, due to a quirk in the Matrix library (I believe).

The message you're sending is not recognised by default. You could either write a plugin for it (if you'd like I can help you out with that), or you can use one of the message formats that are supported by default. For instance, something like this:

 curl -X POST 127.0.0.1:8020/hook/xxxxx -H "Content-Type: application/json" -d '{"text":"hello", "username": "user", "icon_url": "https://url.to.avatar/image.png"}'

The username and icon_url fields are optional.

The formats supported by default can also be found here: https://github.com/geluk/matrix-webhook-gateway/blob/master/src/webhooks/formats.ts

[horstepipe]

ahh thank you very much! Let me take a look at writing a plugin and if you don't mind get back to you.

Best regards!

[horstepipe]

so basically what I need is this: curl -X POST https://webhook.example.com/mywebhookurl -H "Content-Type: application/json" -d '{"movie":"$1","year":"$2","library":"$3"}'

https://emby.media/community/index.php?/topic/86453-flexible-customisable-webhooks-with-emby-scripter-x/&do=findComment&comment=1071582

[geluk]

Here's a simple plugin that will format the above content into a Matrix message.

You can modify the username, icon, and text fields however you like. I don't have any documentation for the plugin API yet, but if you need help changing anything, feel free to ask.

import { assertType } from 'typescript-is';
import { PluginBase, WebhookMessage } from '../../src/pluginApi/v2';
import * as f from '../../src/formatting/formatting';

interface EmbyWebhook {
  movie: string;
  year: string;
  library: string;
};

export const format = 'emby';

export default class EmbyPlugin extends PluginBase {
  async init(): Promise<void> { }

  async transform(body: unknown): Promise<WebhookMessage | undefined> {
    const emby = assertType<EmbyWebhook>(body);
    return {
      username: 'Emby',
      icon: {
        url: 'https://emby.media/notificationicon.png'
      },
      text: f.fmt(
        'Movie ',
        f.quote(emby.movie),
        ' (',
        emby.year,
        ') added to library: ',
        emby.library
      ),
    };
  }
}

[horstepipe]

thank you very much!!! I'll try it out asap

[horstepipe]

humm I think I'm currently a little slow. Would you mind taking a look here, as it is probably not an issue about your gateway.

just if you don't mind and have some minutes!

https://emby.media/community/index.php?/topic/86453-flexible-customisable-webhooks-with-emby-scripter-x/

[geluk]

Based on your logs, it looks like the plugin wasn't executed. I realised that I forgot to mention this, but you should postfix the webhook URL with the name of the plugin (in this case, emby). For example, if your webhook looks like this:

https://webhook.example.com/hook/bt05xpx7splnmw9t192rf5j3zx3i6jc6jnbvhrij

Then you should postfix it with /emby like so:

https://webhook.example.com/hook/bt05xpx7splnmw9t192rf5j3zx3i6jc6jnbvhrij/emby

The webhook should also be dropped into your plugin directory. Any name is fine, as long as it ends with .ts.

If the plugin has loaded successfully, you will see the following message on startup (you may need to raise the log level by running the webhook gateway at a higher verbosity: node entry.js -v) :

2021-10-30 20:13:29.645  DEBUG [webhook-srv] Loading plugin emby.ts from <plugin hash>.js

[horstepipe]

thank you very much for taking the time to troubleshoot my problem. The things you suggested were unfortunately already true.

Here is a full output of the gateway log:

2021-10-31 17:34:35.500.000      INFO   [webhook-srv]           Matrix bridge running on 127.0.0.1:8023
2021-10-31 17:34:35.528.000      WARN   [bridge]                [-] POST http://127.0.0.1:8008/_matrix/client/r0/register (AS) HTTP 400 Error: "{\"errcode\":\"M_USER_IN_USE\",\"error\":\"User ID already taken.\"}"
2021-10-31 17:34:35.602.000      INFO   [webhook-srv]           Loaded plugins: emby
2021-10-31 17:34:35.605.000      INFO   [webhook-srv]           Web server running on 127.0.0.1:8020
2021-10-31 17:35:01.737.000      WARN   [bridge]                [-] POST http://127.0.0.1:8008/_matrix/client/r0/register (AS) HTTP 400 Error: "{\"errcode\":\"M_USER_IN_USE\",\"error\":\"User ID already taken.\"}"
2021-10-31 17:35:02.199.000      ERROR  [webhook-srv]           Failed to handle webhook invocation:
 Error  Cannot read properties of undefined (reading 'formatPlain')
error stack:
• formatting.ts:26 toPlain
    src/formatting/formatting.ts:26:24

• formatting.ts:231 formatPlain
    src/formatting/formatting.ts:231:23

• formatting.ts:26 toPlain
    src/formatting/formatting.ts:26:25

• formatting.ts:36 formatPlain
    src/formatting/formatting.ts:36:30

• formatting.ts:26 toPlain
    src/formatting/formatting.ts:26:25

• MatrixBridge.ts:53 <anonymous>
    src/bridge/MatrixBridge.ts:53:19

• MatrixBridge.js:27 <anonymous>
    src/bridge/MatrixBridge.js:27:71

This is the test curl I'm using:

curl -X POST 127.0.0.1:8020/hook/xxxxx/emby -H "Content-Type: application/json" -d '{"movie":"Testfilm","year":"1999","library":"aktuelle Filme"}'

[geluk]

You're welcome! Unfortunately I can't reproduce the issue locally, but I suspect that something is going wrong with reading the webhook POST content, because it does try to format the data.

Could you modify the plugin like so, and let me know what it logs?

import { assertType } from 'typescript-is';
import { PluginBase, WebhookMessage } from '../../src/pluginApi/v2';
import * as f from '../../src/formatting/formatting';
import logger from '../../src/util/logger';

interface EmbyWebhook {
  movie: string;
  year: string;
  library: string;
}

export const format = 'emby';

export default class EmbyPlugin extends PluginBase {
  async init(): Promise<void> {}

  async transform(body: unknown): Promise<WebhookMessage | undefined> {
    logger.info('Request body: ', body);
    const emby = assertType<EmbyWebhook>(body);
    return {
      username: 'Emby',
      icon: {
        url: 'https://emby.media/notificationicon.png',
      },
      text: f.fmt(
        'Movie ',
        f.quote(emby.movie),
        ' (',
        emby.year,
        ') added to library: ',
        emby.library,
      ),
    };
  }
}

[Dual-0]

@horstepipe I've created a emby.ts plugin:

import { is } from 'typescript-is';
import { PluginBase, WebhookMessage } from '../../src/pluginApi/v2';
import {
    a,
    strong,
    fmt,
} from '../../src/formatting/formatting';

type movie = {
    movie: string;
    year: string;
    imdb: string;
    library: string;
};
type series = {
    series: string;
    imdb: string;
    library: string;
};
type season = {
    season: string;
    series: string;
    imdb: string;
};

export const format = 'emby';

export default class EmbyPlugin extends PluginBase {
  // This function will be executed once, on startup.
  async init(): Promise<void> {
    this.logger.info('emby plugin starting up');
  }

  // This function will be executed every time a webhook with a matching
  // format is posted. It should either return a `WebhookMessage`, if the
  // webhook is to be executed, or `undefined`, if the webhook is to be
  // rejected.
  async transform(body: unknown): Promise<WebhookMessage | undefined> {
    // You can make use of 'typescript-is' to perform runtime type checks on
    // input data. This makes it easy to reject invalid webhooks.
    if (is<movie>(body)) {
        var link = 'https://www.imdb.com/title/' + body.imdb;
        var titel = body.movie + ' (' + body.year + ')';
        return {
            // username: 'Emby Bot',
            text: fmt(
                'The Movie ',
                strong(a(link,titel)),
                ' was added to the ',
                body.library,
                ' library',
            ),
          };
    }
    if (is<series>(body)) {
        var link = 'https://www.imdb.com/title/' + body.imdb;
        return {
            // username: 'Emby Bot',
            text: fmt(
                'The Series ',
                strong(a(link,body.series)),
                ' was added to the ',
                body.library,
                ' library',
            ),
          };
    }
    if ((is<season>(body) {
        var link = 'https://www.imdb.com/title/' + body.imdb;
        return {
            // username: 'Emby Bot',
            text: fmt(
                'Season ',
                body.season,
                ' was added to ',
                strong(a(link,body.series)),
            ),
          };

    } else {
        this.logger.warn('Invalid webhook');
        this.logger.warn(body);
        return undefined;
    }
  }
}

webhook.sh shell script for sending webhooks:

#!/bin/bash

# Variables
imdb_pattern='^tt[0-9]*$'
webhook_url='https://webhook.domain.com/hook/pvp***f53'

webhook() {
    curl -X POST "$webhook_url/emby" -H "Content-Type: application/json" -d "$1"
}

if [[ $1 == Movie ]] && [[ "$4" =~ $imdb_pattern ]]; then
    webhook "{\"movie\":\"$2\",\"year\":\"$3\",\"imdb\":\"$4\",\"library\":\"$5\"}"
elif [[ $1 == Series ]] && [[ "$3" =~ $imdb_pattern ]]; then
    webhook "{\"series\":\"$2\",\"imdb\":\"$3\",\"library\":\"$4\"}"
elif [[ $1 == Season ]] && [[ "$4" =~ $imdb_pattern ]]; then
    webhook "{\"season\":\"$2\",\"series\":\"$3\",\"imdb\":\"$4\"}"
fi

example parameters:

bash webhook.sh "Movie" "Django Unchained" "2012" "tt1853728" "My Movies"
bash webhook.sh "Series" "Squid Game" "tt10919420" "My Series"
bash webhook.sh "Season" "1" "Squid Game" "tt10919420"

give example messages:

@geluk any advices for the emby.ts plugin?

[geluk]

@Dual-0 Once the next version of the webhook gateway comes out, there will be a few more formatting functions to further simplify the text formatting that you're doing here, but other than that I don't have any significant feedback. What you're showing here is exactly what I hoped people would be able to do, which is to create their own plugins tailored to their own use cases, so I'm happy to see that it's working.

There's one thing I'll clarify; you can either use type or interface for your type definitions. For the purposes of making webhook plugins, there are no real differences between these two, so use whatever you like (in fact in my examples I've used both as well). It is conventional to use uppercase names for types to distinguish them from variables, but as long as you can read the code, that doesn't really matter here.

Lastly, I'd be interested in hearing about your experiences writing the plugin. Did you run into any issues? Was there anything in the process that's unclear or could be made more ergonomic? If you have any feedback here, I'll use that to improve the process for future users.

[horstepipe]

thank you very much dual-o for sharing! Unfortunately, i'm still getting the same error. I guess I need to try the docker installation as it is more up to date than the normal installation.

[geluk]

@horstepipe The Docker version has a :dev tag which tracks the latest development version. I don't currently have a way of automatically distributing the latest version of the normal version, but I've uploaded a recent build here: http://geluk.io/public/dl/webhook-gateway-b2f5eda.tar.gz

[Dual-0]

@horstepipe: I use it without docker. just the "latest" tagged release + node + pgsql. my systemd file:

[Unit]
Description=matrix-webhook-gateway daemon
PartOf=matrix-synapse.service
Requires=matrix-synapse.service
After=matrix-synapse.service

[Service]
Type=simple
WorkingDirectory= /opt/webhook-gateway
ExecStart=node entry.js -c "/opt/webhook-gateway/gateway-config.yaml" -a "/etc/matrix-synapse/appservices/appservice-webhook-gateway.yaml"
RestartSec=20
Restart=always

[Install]
WantedBy=matrix-synapse.service

[horstepipe]

@horstepipe: I use it without docker. just the "latest" tagged release + node + pgsql. my systemd file:

[Unit]
Description=matrix-webhook-gateway daemon
PartOf=matrix-synapse.service
Requires=matrix-synapse.service
After=matrix-synapse.service

[Service]
Type=simple
WorkingDirectory= /opt/webhook-gateway
ExecStart=node entry.js -c "/opt/webhook-gateway/gateway-config.yaml" -a "/etc/matrix-synapse/appservices/appservice-webhook-gateway.yaml"
RestartSec=20
Restart=always

[Install]
WantedBy=matrix-synapse.service

thanks. okay if it also works via non docker for you, something else seems to go wrong here. I'll reinstall and reconfigure the whole thing and see if this helps.

[horstepipe]

which node version are you running? I do 16.13.0

[horstepipe]

another thing, from your config it looks like the webhook is on an external server? Here it is the same server. Is there anything which could be problematic about that?

[horstepipe]

okay forget it. IT IS WORKING NOW, yeha! :-) I deleted the cache and workdir directories at the plugins folders, now it works as intended.

Thank you very very much to you guys for helping me setting this up!