Message-based transport of sonar data

[lars-t-hansen]

Without changing sonar we can move from disk-based to message-based transport. This will aid deployment on many systems and I think it's right to do this sooner rather than later.

Currently sonar prints its output to stdout and the shell script writes it to disk. I think that instead, the shell script should pipe it into a program I've written (*) called exfiltrate that pushes it (by HTTP POST currently but could be MQTT, see #285) to an agent off-system, we'd want to run this agent on the VM that's hosting the website. The agent would store the data locally and would run analyses on the data locally, this way we'd get rid of the cron jobs running on moneypenny and login-1.fox.

This system is a little more brittle than the shared disk but in the long run it's what we want anyway, so let's just bite the bullet and work on resilience in the implementation.

I think I'd like to do this before I do anything about moving sonar out of my account on the ML nodes, hence making this block M1.

(*) The program is working but it is missing a couple of optimizations and some resilience, both will be important.

The basic transport is working now. Remaining task list:

On the sending side

• [x] exfiltrate.go: Maybe: Batching of data
• [x] exfiltrate.go: Maybe: retry sending when something goes wrong
• [x] Infra: Change sonar.sh to send data rather than / in addition to storing to disk
• [x] Even for the very first version there needs to be some kind of secret to authenticate the data

On the receiving side

• [x] infiltrate.go: Try to avoid logging storms?
• [x] infiltrate.go: Some reliable way of stopping the service (eg SIGHUP)
• [x] Infra: ingestion of data

Transfer the ML nodes to the new system:

• [x] spin down the new prototype cronjob on ml4 and spin up the original one
• [x] setup the latest infiltrate on the sonar vm, note authentication
• [x] Get authentication right on the sending side too
• [x] create a proper new cronjob and scripts for the ml nodes, segregating things properly
• [x] spin up the new cronjobs, data will be stored in files on the shared filesys but also sent to the sonar vm
• [x] monitor this for a little while, but we've tested already so should be ok
• [x] eg tomorrow, files should be similar on the two hosts, but equality is not likely both because record ordering can change and because measurements and timestamps are obtained twice and will not be equal
• [x] set up prototype analysis cronjobs on the sonar vm
• [x] test that analysis files look ok, ie more or less match the ones generated on moneypenny; can copy over to staging server probably?
• [x] give it a couple days to stabilize
• [x] implement file copy on the sonar vm of the analysis results to the web server (can be scp...)
• [x] copy over all the missing data files (at least thru dec 6 but maybe later if we find bugs) from the shared disk to the sonar vm, these will all be at least a few days old
• [x] #301
• [x] get mail working on the sonar VM, we need MAILTO to work in general
• [ ] some kind of method for making sure the server is always up
• [x] figure out what happens with server log messages - they need to be visible somehow
• [x] figure out how to get info about violators and hogs into the RT queue, now that MAILTO is not going to work well for that
• [x] spin down the analysis jobs on moneypenny
• [x] make upload config per-cluster, b/c it allows for better experimentation / dev
• [x] change upload config from staging to production
• [x] AT THIS POINT, WE ARE RUNNING ON EXFILTRATED DATA
• [x] remove output-to-file from the cronjobs on the ML nodes
• [x] AT THIS POINT, INFILTRATE IS SINGLE POINT OF FAILURE FOR DATA COLLECTION
• [ ] let it stabilize a bit
• [ ] the files on the shared disk can be archived and the directory removed
• [x] document the hell out of everything
• [x] remove pointless files from the production/ml-nodes directory (most of them)
• [x] figure out a backup solution for the files on the VM

Once that has baked for a while, we can do the same on Fox, though it'll likely be a little simpler

[lars-t-hansen]

To bring and keep the ingestion service up it'll be some systemd thing, see https://chris-vermeulen.com/using-systemd-to-run-a-simple-process-and-keep-it-up/ and https://wiki.archlinux.org/title/systemd/User for starters though some thinking is required to find the best form of it. Is it a user service or a system service?

[lars-t-hansen]

Re "Figure out how to get info about violators and hogs into the RT queue, now that MAILTO is not going to work well for that - we want MAILTO for cron failures, not for desired output", the most sensible thing to do would probably be to just add a -mailto option to naicreport, -mailto itf-ai-support@usit.uio.no, this would email all non-error output to that address with a suitable from: and subject line.

[lars-t-hansen]

A complication with the educloud VM is that though it is "open to the world" the available ports for TCP are very, very restricted and the list is controlled by Central Services. In particular, the ports I've been using are verboten. For the time being I can work around that using the ports that are left open for Oracle use. In the slightly longer term it'll be necessary to set up nginx proxies (probably) to forward requests to the services that are open on the machine but not visible from the outside.

[lars-t-hansen]

See https://github.com/NAICNO/Jobanalyzer/pull/331.

[lars-t-hansen]

I'm spinning off the systemd task and the remove-the-old-files task into separate bugs. Otherwise we're done: sonar on the ML nodes now stores no data on local disk and instead exfiltrates everything to naic-monitor.