search

CLIP-HPC / goslmailer

GoSlurmMailer - drop in replacement for default slurm MailProg. Delivers slurm job messages to various destinations.
MIT License
42 stars 6 forks source link
discord hpc mail matrix mattermost msteams notifications slack slurm telegram

readme

goslmailer

News & Info

Discussion forums have been enabled for this project, feel free to share your thoughts and ideas there.

v2.7.1

  • In v2.7.0 dumping of configuration at goslmailer/gobler startup was removed.
  • Now it can be selectively enabled with debugconfig: true parameter in config file

v2.7.0

v2.5.0

New: In addition to json, now supports TOML configuration files (.toml file suffix)

BREAKING CHANGE: qosmap changed its format from map[int]string to map[string]int. Adopt the goslmailer config file accordingly.

 "qosmap": {              # map of your sites configured QoSes, with their timelimits (seconds), used for hint generation
-   "3600": "RAPID",
-   "28800": "SHORT",
-   "172800": "MEDIUM",
-   "1209600": "LONG"
 "qosmap": {              # map of your sites configured QoSes, with their timelimits (seconds), used for hint generation
+   "RAPID": 3600,
+   "SHORT": 28800,
+   "MEDIUM": 172800,
+   "LONG": 1209600
 }

Drop-in notification delivery solution for slurm that can do:

Description

Goslmailer (GoSlurmMailer) is a drop-in replacement MailProg for slurm.

With goslmailer configured as as the slurm mailer,

MailProg                = /usr/bin/goslmailer

it provides users with the ability to specify a comma-separated list of receivers [connector:]target in the sbatch --mail-user switch to select where the messages will be sent out (similar to URI scheme).

e.g.

sbatch --mail-type=ALL --mail-user="mailto:useremailA,msteams:usernameB,telegram:NNNNNNN,usernameC"

To support future additional receiver schemes, a connector package has to be developed and its configuration block present in configuration file.

Currently available connectors:

See each connector details below...

If you would like to contribute to this project by developing a new connector, here you can find a heavily annotated connector boilerplate (fully functional) to help you get started.

Building and installing

Option 1. Download latest precompiled binaries here

Unpack, follow instructions

Option 2. Build

Quick version, without end to end testing

git clone https://github.com/CLIP-HPC/goslmailer.git
make test
make build
make install

Slightly more involved, with end to end testing:

Prerequisites:

  1. generated RSA keypair (passwordless) (ssh-keygen -t rsa)
  2. ssh $USER@localhost must work without password

Known caveats:

try:

mkdir $HOME/.secret
ssh-keygen -b 1024 -t rsa -f id_rsa -P "" -f $HOME/.secret/id_rsa
touch ~/.ssh/authorized_keys
cat $HOME/.secret/id_rsa.pub >>  ~/.ssh/authorized_keys 
# downloads endly binary and runs endly tests
make

Install

goslmailer

gobler

tgslurmbot

matrixslurmbot

discoslurmbot

Spooling and throttling of messages - gobler service

In high-throughput clusters or in situations where job/message spikes are common, it might not be advisable to try to send all of the incoming messages as they arrive. For these environments goslmailer can be configured to spool messages from certain connectors on disk, to be later processed by the gobler service.

gobler is a daemon program that can be configured to monitor specific spool directories for messages, process them and send out using the same connectors as goslmailer.

On startup, gobler reads its config file and spins-up a connector monitor for each configured spool directory.

connector monitor in turn spins up 3 goroutines: monitor, picker and numSenders x sender.

Artistic sketch of the system described above

Sketch of the architecture

Connectors

connector spooling/throttling capable (gobler)
mattermost yes
discord yes
matrix no
telegram yes
msteams yes
mailto no
slack yes

default connector

Specifies which receiver scheme is the default one, in case when user didn't specify --mail-user and slurm sent a bare username.

"defaultconnector": "msteams"

mailto connector

Mailto covers for original slurm e-mail sending functionality, plus a little bit more. With connector parameters, you can:

known caveats

# Local configuration for Mutt.
set content_type="text/html"

See annotated configuration example

mattermost connector

Prerequisites for the mattermost connector:

  1. Create mattermost bot, instructions here.
  2. OPTIONAL/CONVENIENT: Start the bot daemon service mattermostbot (example config file).
    • This is used to deliver --mail-user=mattermost:chatId messages, can also be looked up by the user in the Info window
  3. If the bot is running, it will wake up on the configured triggerString and send the user a private message with slurm job submission instructions
  4. Put mattermost block in goslmailer.conf (and optionally gobler.conf if spooling is used)

Mattermost card

Additional helpful sources:

discord connector

Prerequisites for the discord connector:

  1. a discord bot must be created and
  2. the bot daemon service discoslurmbot must be running (example config file).
  3. once the bot is running, it will wake up on the configured triggerString and send the user a private message with slurm job submission instructions

Discord Bot setup

  1. User settings -> Advanced -> Developer mode ON
  2. Discord developer portal -> New Application -> Fill out BotName
  3. Once the application is saved, select Bot from left menu -> Add Bot -> message: "A wild bot has appeared!"
  4. Left menu: OAuth2 -> Copy Client ID
  5. Modify this url with the Client ID from 4. and open in browser: https://discord.com/api/oauth2/authorize?client_id=<CLIENT-ID>&permissions=8&scope=bot
  6. "An external application BotName wants to access your Discord Account" message -> Select server -> Continue
  7. Grant Administrator permissions -> yes/no/maybe ? -> Authorize
  8. Discord developer portal -> Select BotName -> Bot menu -> Reset Token -> Copy and Save, to be used in discoslurmbot.conf

Or follow this tutorial

Discord card

telegram connector

Sends 1on1 or group chat messages about jobs via telegram messenger app

Telegram card

Prerequisites for the telegram connector:

  1. a telegram bot must be created and
  2. the bot daemon service tgslumbot must be running.

Site admins can create a telegram bot by messaging botfather.

Once the bot is created, you will receive a bot token. Place the bot token in the goslmailer/gobler config file in the telegram connector section (see example below).

Start the tgslurmbot binary that serves as the bot.

When the chat/group chat with the bot is initiated and/or the bot receives a /start command, he will reply with a chat-specific --mail-user=telegram:nnn message which the user can use in his slurm job scripts to get the job messages.

See annotated configuration example

matrix connector

Sends messages to user-defined Matrix rooms.

Matrix

The Matrix connector can send messages through a pre-defined user that you manually join into channels. In addition, you can setup matrixslurmbot to control this pre-defined user and have him join rooms upon invitation.

The first thing you need is to add the proper parameters to the connectors/matrix section of the config file.

To obtain the access token, you can run this command using curl:

curl -XPOST -d '{"type":"m.login.password", "user":"@myuser:matrix.org", "password":"mypassword"}' "https://matrix.org/_matrix/client/r0/login"

Which should get you a response akin to this one, containing the token:

{
    "user_id": "@myuser:matrix.org",
    "access_token": "syt_dGRpZG9ib3QXXXXXXXEyQMBEmvOVp_10Jm93",
    "home_server": "matrix.org",
    "device_id": "DMIIOMEYBK",
    "well_known": {
        "m.homeserver": {
            "base_url": "https://matrix-client.matrix.org/"
        }
    }
}

Once you've done this, goslmailer will be able to send messages to any channel the configured user is a member of with a command like this:

sbatch --mail-type=ALL --mail-user='matrix:!VEbreVXXXkmLWjeGPi:matrix.org' --wrap "ls"

Where !VEbreVXXXkmLWjeGPi:matrix.org is the ID of the channel you want to send the message to. You can obtain this ID from the Settings>Advanced section of the room.

Using the bot

Instead of manually having the pre-defined user join certain rooms, you can use matrixslurmbot to control this user and have it join rooms automatically upon invitation. For this you just need to start the matrixslurmbot, pointing it to a config file identical to the one before using (you could strip it down to contain only the 'matrix' section).

While the bot is running, any user can invite it to join a channel (e.g. /invite @mybotuser:matrix.org). The bot will join the channel and post the --mail-user:... argument to be used when submitting jobs in order to receive a message in that channel.

In the event that the pre-defined user is alone in a channel (i.e. all normal users left), the bot will make the pre-defined user leave and forget the channel.

msteams connector

Sends a message to a preconfigured ms teams channel webhook.

MS Teams card

Since MS Teams does not provide with the option to send messages to users directly, only to channel webhooks, we have devised a way using MS Power Automate framework to pick up messages from this one configured sink channel and deliver them via private 1on1 chats to the recipient user.

Users listed in the --mail-user=msteams:userA,msteams:userB will be sent as adaptive card mention entity. A MS Power Automate workflow monitors the configured sink channel, parses the received adaptive card jsons, locates the mention entity and delivers to it the copy of the message via private chat.

See annotated configuration example

slack connector

Sends messages to direct messages or channels in Slack using its Web API and slack-go.

Prerequisite for the slack connector:

  1. A Slack App must be created

Slack App setup

  1. Go to the Slack App portal. Give your app a name (such as "SlurmBot"), and choose the workspace you want to use the app in. You must be allowed to add apps to this workspace.
  2. Click on OAuth & Permissions on the sidebar. Scroll down to Scopes and click on Add an OAuth Scope. Add chat:write and im:write.
  3. Scroll back up to OAuth Tokens for Your Workspace and copy the Bot User OAuth Token. It should look something like xoxb-5555555555555-5555555555555-XXXXXXXXXXXXXXXXXXXXXXX. This will be used in your goslmailer config (goslmailer.conf or goslmailer.toml), and gobler config if used.

Slack card

Notes

The bot will be able to send to any channel it's in. Make sure it is not added to channels it shouldn't send messages in (such as an announcments channel). You can add the bot to a channel by pinging it (send a message with just @SlurmBot, for example) and then click on Add to Channel.

Gotchas

msteams

references

msteams