mvo5 / unattended-upgrades

Automatic installation of security upgrades on apt based systems
GNU General Public License v2.0
285 stars 77 forks source link

Webhook support with apprise #325

Closed papamoose closed 2 months ago

papamoose commented 2 years ago

Adds webhook support. This is a rethink on pull request #307, which I've now closed.

It uses the python module apprise to allow a user to specify any number of webhooks.

This commit compiles a dictionary with what actions u-u has taken and POST's the result as json to the webhooks specified.

Example of the resulting json POST'd.

{
  "hostname": "zeus.example.com",
  "reboot_flag": true,
  "hold_flag": false,
  "result": "SUCCESS",
  "packages": {
    "upgraded": [
      "alsa-ucm-conf",
      "cloud-init",
      "libdrm-common",
      "libdrm2",
      "libnetplan0",
      "libnss-systemd",
      "libpam-modules",
      "libpam-modules-bin",
      "libpam-runtime",
      "libpam-systemd",
      "libpam0g",
      "libprocps8",
      "libsystemd0",
      "libudev1",
      "libudisks2-0",
      "linux-firmware",
      "netplan.io",
      "open-vm-tools",
      "procps",
      "python-apt-common",
      "python3-apt",
      "python3-update-manager",
      "snapd",
      "systemd",
      "systemd-sysv",
      "systemd-timesyncd",
      "thermald",
      "udev",
      "udisks2",
      "update-manager-core"
    ]
  }
}

@simon-ourmachinery

mvo5 commented 2 years ago

Thanks for this pull request. Let me think about this, I like the idea and I understand in todays world webhooks are much more important than sending a summary mail :) Here are some preliminary thoughts: One thing I'm unsure about is the use of apprise, AFAICT it's not packaged yet for Debian or Ubuntu so I would not be able to include it in the default packages (yet). I would like to make sure your use-case gets supported though. I wonder if you would benefit from having a plugin mechanism in unattended-upgrades, i.e. something like /usr/share/unattended-upgrades/plugins (and one version in /etc) that is imported by u-u and that will call pre/post upgrade hooks with essentially the same information as "send_summary_mail" has? Wdyt?

papamoose commented 2 years ago

I see how apprise not having a debian package would a roadblock. I thought it was nice as u-u wouldn't have to maintain any specifics on any particular webhook.

A plugin system like you describe sounds like a great idea! An ability to import send_summary_mail data to a plugin would definitely allow for the greatest flexibility and not require u-u to deal with any nuance of any webhook in particular, but instead stay focused on its core goal.

Additionally, it would allow for more customization by the user. Instead of only being able to send json to a webhook the plugin could customize the message to display in a more user friendly way based on the webhook used.

How do you imagine a user would specify the webhook url(s) and how would they be used in the context of the plugin?

mvo5 commented 2 years ago

@papamoose Thanks for your thoughts on a plugin system. Python makes this fairly easy and it would allow some flexiblity, we could even ship some example plugins (e.g. one that uses apprise :)

I drafted some ideas in https://paste.ubuntu.com/p/jQ7fzvH253/ - obviously very incomplete but hopefully enough to get the idea. The user could install plugins into a location (TBH) like /etc/unattended-upgrades/plugins/ and the plugin can be any class that provides a "summary()" method with the right signature (also TBD, probably some dict for easier extensibility). WDYT?

mvo5 commented 2 years ago

Fwiw, I pushed a slightly more elaborate version of the plugin idea into https://github.com/mvo5/unattended-upgrades/compare/plugins?expand=1 still needs some tweaks as it e.g. does not capture right now if a reboot is required (and some other things like hold pkg handling) or not but that should be pretty trivial to add.

papamoose commented 2 years ago

Thanks. What you posted on launchpad looks good. I'll take a look at your most recent update soon. I'm switching jobs and moving within the next month so I expect to have little time to dedicate to this. I'm not ignoring your on purpose. :) Your work on the plugin system is much appreciated.

mvo5 commented 2 years ago

Thanks @papamoose - good luck with your job switch and the moving. I will keep working on this, probably a bit slower from next week on. Given that it's a public API that will need to be supported forever I don't want to rush things and be a bit careful about naming of e.g. the dict keys etc. I am also wondering if I should make it even more generic by just calling any executable in the plugin path and sending the data via json to stdin of the process. Makes the plugins themselfs slightly more complex (as they now need to read from stdin and de-serialize) but one can write in any language (hello /bin/sh) this way :)

a-detiste commented 2 years ago

Hi, this new plugin system is awesome. The python interface is fine. The "json / stdin" would be more limitatign in the future I guess; if the PluginManager do more stuff with the plugin objects.

mabed-fr commented 2 years ago

Hello

What method will be used?

GET? PUSH? PUT

Will it be possible to configure the headers to be sent?

Will there be more than one variable for the content of the message?

mvo5 commented 2 years ago

Hello

What method will be used?

GET? PUSH? PUT

Will it be possible to configure the headers to be sent?

Will there be more than one variable for the content of the message?

Thanks for your interest. The current thinking is that unattended-upgrades will not by itself do the webhook call but instead provide a plugin interface that gives the plugin enough information to perform the action. So you have total flexibility for get/post/put. We will probably also push an example plugin (this apprise one looks very nice for example).

mabed-fr commented 2 years ago

Thank you for your explain.

I am missing a point to clarify. Who will be the trigger for this webhook / plugin?

Mabed

papamoose commented 2 years ago

As @mvo5 said. The goal has changed. u-u will only provide a plugin system which will probably pass a dictionary with the summary of changes to any and all plugins that is on the user to write.

As it stands now this commit only passes the summary data as text (in json format) to whatever webhooks apprise supports. I don't intend for it to do more than that at this time.

mvo5 commented 2 years ago

Fwiw, I worked a bit more on this in https://github.com/mvo5/unattended-upgrades/compare/plugins?expand=1 and I'm reasonable happy with the result. The only thing I'm still mulling over is if the argument for the new postrun() function should be a simple python dict with json-like keys as it is right now (which will make the webhook thing very straightforward) or if instead I should use a python class/dataclass - the upside of this approach is that it's more python-ish and that mypy could ensure a certain level of type-safety when working with the data but it's more cumbersome in the plugin side if most plugins use this in a json(ish) context. Let me know if someone as opinions about this question please :)

And of course the next step is also to ship this nice apprise work as a advanced example or even as a default.

a-detiste commented 2 years ago

The objec-ified way is cleaner ...

papamoose commented 1 year ago

Finally had some time to take a look at this (https://github.com/mvo5/unattended-upgrades/compare/plugins?expand=1).

I was able to get a rough draft, minimal viable product, of the apprise json notifier plugin working.

  1. Re: postrun python dict vs class/dataclass. I'll use either. Using the dict was easy and a low barrier to entry. As long as there is an example included on how to use I doubt it will make much difference to anyone writing a plugin. Meaning @mvo5 should choose what they prefer.
  2. The plugin system seems to be working as expected!
  3. I'm taking advantage of import apt_pkg so the user can set variables. Moving these variables into a dedicated file worked great. /etc/apt/apt.conf.d/51uu-apprise.
  4. I also imported and copied the logging functions from u-u into my script so that it logs in the same way u-u does. It would be nice to get log_once() without needing to copy it into the plugin.
    import logging
    logged_msgs = set()  # type: AbstractSet[str]
    def log_once(msg):
    # type: (str) -> None
    global logged_msgs
    if msg not in logged_msgs:
        logging.info(msg)
        logged_msgs.add(msg)  # type: ignore

I'm hoping to have a bit more time now so getting the plugin into a state that others can look at shouldn't be too much longer now.

papamoose commented 1 year ago

@mvo5 https://github.com/papamoose/unattended-upgrades/blob/plugins-apprise/examples/plugins/apprise.py

This is just your plugins branch with a WIP of my plugin. I want to rename it and some of the functions inside it but it works as is now. I may end up refactoring it a bit too as it follows the pattern mail uses inside u-u which I no longer feel obligated to follow and could clean it up some.

I'd be happy to test a objectified version, as I did find I was passing result around more than I thought was necessary.

stephen99scott commented 1 year ago

@papamoose @mvo5 This is a very useful feature! I have a Raspbian Buster server natively running u-u@1.11.2, so I back-ported plugins support to this release (https://github.com/mvo5/unattended-upgrades/compare/1.11.2...stephen99scott:unattended-upgrades:plugins-buster?expand=1).

a-detiste commented 1 year ago

I have a Raspbian Buster server

This I will need this too for my 1300 brand new physicial Buster servers :-)

papamoose commented 11 months ago

Not sure where we left off with this... but I've been running this for the past year and have been very happy with it.

PR issued on the plugins branch: https://github.com/mvo5/unattended-upgrades/pull/349 Feel free to deny if this isn't how you want to proceed. I'm just getting the conversation started again around this. :)

mvo5 commented 8 months ago

I finally managed to spent a bit of time on this again and I am quite happy with my current code in https://github.com/mvo5/unattended-upgrades/pull/355 - if someone could double check and test that would be super appreciated.

(and SORRY that this took me forever to get to!)

JonnyBDev commented 4 months ago

Hey guys,

just wanted to keep the conversation going. Currently in the process of using unattended-upgrades and we'd love to see a feature like this! We really need Webhooks for the management of our servers.

I can see that the issue #355 is in the status "Merged". Am I missing something here or is this already implemented? A bit confusing, that this PR is still open - even if it's for an older iteration of the system.

Would appreciate some feedback :) Thanks!

papamoose commented 4 months ago

@mvo5 has added plugin support. If you wish to add my apprise plugin you would need u-u v2.11+.

Then you can take a look my repo that contains notification plugin: https://github.com/papamoose/unattended-upgrades-plugin-webhook

I think this MR should be closed. What do you think @mvo5?

mvo5 commented 4 months ago

@mvo5 has added plugin support. If you wish to add my apprise plugin you would need u-u v2.11+.

Then you can take a look my repo that contains notification plugin: https://github.com/papamoose/unattended-upgrades-plugin-webhook

I think this MR should be closed. What do you think @mvo5?

Yeah, I am fine closing it in it's current form. I would like to include a link to your plugin repo in the readme, I think that would be nice for people who need this functionaltiy. Or even include it in the examples/ dir (if it does not change very much) :)

papamoose commented 4 months ago

I would like to include a link to your plugin repo in the readme, I think that would be nice for people who need this functionality.

Sure!

Or even include it in the examples/ dir

I'm of two minds on this:

  1. putting it in the examples directory will distribute the code out to the people that want to use it. Which is excellent!
  2. It seems like a plugin should be a separate package (willing to be convinced otherwise). Though, I'm not familiar with the process in getting a package into Debian and I assume that would take a while to do.

Therefore, it seems like putting the plugin in the examples directory is a great solution for right now, with a longer term goal of getting the package added to Debian.

What do you think?

(if it does not change very much) :)

I don't imagine it would need to change much. Since I've written it I don't think I've touched the code outside the time you added plugin support. Worst case, you could kick it out if it becomes a problem and just point at my repo.

papamoose commented 2 months ago

I'm going to close this in favor of adding this as an example in #369.