Add webhook support - Githubissues

papamoose commented 2 years ago

I've added webhook support. Most of the testing was done sending to Discord and a local python webserver that displays whatever is in POST.

Used def send_summary_mail() as a guide.
left out sending dpkg_log and mem_log.
- The variables still get passed around, but don't get used.
The json data structure should be given a second look before it becomes permanent.
2000 character limit for no other reason than that is what Discord's limit is. If it's over 2000, I overwrite the 'package' key saying it's too long to include and send the notification anyways.
I'm currently testing this code on some of my personal servers. Maybe others can test as well.
This uses python3-requests, so the package would grow a dependency. Maybe urllib should be used instead.

The goal is to provide data to web server that can ingest JSON and then do something smart with it.

TODO: Can someone write test code for this or help me write it?

Let me know what you think.

Sample output in Discord:

Spidey Bot
BOT
 — Today at 3:59 PM
{"hostname": "zeus.example.com", "reboot_flag": true, "hold_flag": false, "result": "SUCCESS", "packages": {}}

Spidey Bot
BOT
 — Today at 4:13 PM
{"hostname": "zeus.example.com", "reboot_flag": true, "hold_flag": false, "result": "SUCCESS", "packages": {"upgraded": ["alsa-ucm-conf", "cloud-init", "libdrm-common", "libdrm2", "libnetplan0", "libnss-systemd", "libpam-modules", "libpam-modules-bin", "libpam-runtime", "libpam-systemd", "libpam0g", "libprocps8", "libsystemd0", "libudev1", "libudisks2-0", "linux-firmware", "netplan.io", "open-vm-tools", "procps", "python-apt-common", "python3-apt", "python3-update-manager", "snapd", "systemd", "systemd-sysv", "systemd-timesyncd", "thermald", "udev", "udisks2", "update-manager-core"]}}

After piping to jq:

{
  "hostname": "zeus.example.com",
  "reboot_flag": true,
  "hold_flag": false,
  "result": "SUCCESS",
  "packages": {
    "upgraded": [
      "alsa-ucm-conf",
      "cloud-init",
      "libdrm-common",
      "libdrm2",
      "libnetplan0",
      "libnss-systemd",
      "libpam-modules",
      "libpam-modules-bin",
      "libpam-runtime",
      "libpam-systemd",
      "libpam0g",
      "libprocps8",
      "libsystemd0",
      "libudev1",
      "libudisks2-0",
      "linux-firmware",
      "netplan.io",
      "open-vm-tools",
      "procps",
      "python-apt-common",
      "python3-apt",
      "python3-update-manager",
      "snapd",
      "systemd",
      "systemd-sysv",
      "systemd-timesyncd",
      "thermald",
      "udev",
      "udisks2",
      "update-manager-core"
    ]
  }
}

mvo5 commented 2 years ago

Nice idea! A bit of a meta-comment - u-u runs in some minimal chroots so we try to keep the number of dependencies as low as possible. if "requests" requires an extra deb package (I don't know if it does, this is really a quick drive-by review) it would be lovely if it could a) optional and we just fail gracefully if not there b) using the http stuff included in python itself (and ideally even work with python2 [1])

Thanks! Michael

[1] At least in the past I tried to keep the code compatible so that really old systems can still benefit from u-u in e.g. ESM scenarios and such. But I guess it becomes less and less relevant these days :)

papamoose commented 2 years ago

@mvo5 I think using urllib.requests from python standard lib is do-able and probably a good idea to eliminate the extra dependency. I'll update the patch accordingly.

Re [1]: "Python 2 is unsupported by the Python Foundation since 2020-01-01 and is being removed as of Debian 11 (Bullseye)." https://wiki.debian.org/Python; https://www.python.org/doc/sunset-python-2/ At least on my Ubuntu system the u-u script specifies #!/usr/bin/python3 specifically. Maybe you can provide context I don't have, but it seems python2 support might be unnecessary.

papamoose commented 2 years ago

I've done some investigating and it doesn't seem like urllib.request will not work for most cases. Meaning the requests library would be preferred. I'm not an expert but it seems to come down to support for Connection: keep-alive, which requests provides.

In my testing I was able to POST to my custom web server, but unable to POST to Discord using urllib.request.

I will change the code to gracefully fail if requests is not there and move python3-requests to Suggests: in debian\control.

$ ./show_GET_and_POST_requests.py 8088
INFO:root:Starting httpd...

INFO:root:POST request,
Path: /
Headers:
Content-Length: 113
Host: localhost:8088
User-Agent: Python-urllib/3.8
Content-Type: application/json
Connection: close
Accept-Encoding: gzip, deflate
Accept: */*

Body:
{"content": {"hostname": "hades", "reboot_flag": false, "hold_flag": false, "result": "SUCCESS", "packages": {}}}

127.0.0.1 - - [15/Oct/2021 11:00:55] "POST / HTTP/1.1" 200 -
INFO:root:POST request,
Path: /
Headers:
Host: localhost:8088
User-Agent: python-requests/2.22.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Content-Length: 113
Content-Type: application/json

Body:
{"content": {"hostname": "hades", "reboot_flag": false, "hold_flag": false, "result": "SUCCESS", "packages": {}}}

127.0.0.1 - - [15/Oct/2021 11:00:55] "POST / HTTP/1.1" 200 -

show_GET_and_POST_requests.py

papamoose commented 2 years ago

always and on-change options seem to be working as expected.

{"hostname": "gw.example.com", "reboot_flag": false, "hold_flag": false, "result": "SUCCESS", "packages": {"autoremoved": ["linux-image-5.4.0-81-generic", "linux-modules-extra-5.4.0-81-generic", "linux-headers-5.4.0-86-generic", "linux-modules-5.4.0-81-generic", "linux-image-5.4.0-86-generic", "linux-modules-5.4.0-86-generic", "linux-modules-extra-5.4.0-86-generic", "linux-headers-5.4.0-81", "linux-headers-5.4.0-81-generic", "linux-headers-5.4.0-86"]}}

simon-ourmachinery commented 2 years ago

What is needed to make them be able to push to master? @mvo5 @papamoose

I kind of need this since I wanted to integrate this into our slack communication.

papamoose commented 2 years ago

I've been thinking about how to implement this feature for a bit a now and have rewritten this commit to use Apprise instead which allows for all kinds webhooks. This keeps any specific webhook code out of the UU script.

The branch I think should be used: https://github.com/papamoose/unattended-upgrades/tree/webhook-support-apprise

I can close this pull request and submit a new one. Someone should take a look at the code and in addition to reviewing it, make sure any decisions I've made don't conflict with the project.

simon-ourmachinery commented 2 years ago

I would really like this to be in there.

papamoose commented 2 years ago

Closing in favor of re-implementation in #325.

mvo5 / unattended-upgrades

Add webhook support #307