Move job-runner on to host rather than docker

[bloodearnest]

Currently the job-runner runs inside docker, which causes IO slowness

Instead, we'd like to try run in natively on the TPP host. Python has been installed, and it works, but we need to run it as a service, so it can be managed properly.

The nssm tool can help us here.

Rough design is to set up service with nssm that:

• runs as Seb's user.
• can be restarted by any L2/3 user.
• executable is c:\path\to\python.exe E:\job-runner\windows-service.py or similar (might be an entrypoint)

windows-service.py will:

a) load and set environment variables b) start the the 2 jobrunner process. c) respond to windows service commands as needed.

Open questions:

• can nssm work in the way we want (i.e. running as user)?
• how will we install job-server? As pypi package or checkout?
• is there any specific windows service boilerplate we need beyond nssm?
• logging for new service set up.

[bloodearnest]

Progress

I managed to get this working with a simple python script.

• nssm can send the equivalent of a SIGINT/Ctrl-C, which can be handled in python with KeyboardInterrupt
• was able to get the service to run as a local user
• was able to give my local user permission to do nssm stop opensafely and nssm start opensafely via a medium complexity method using the builtin MMC and the hilariously named 'Snap-ins'. Details here

Made an install.bat script. Batch because the target user to run it is TPP, not our lot. Similar to sebs with some tweaks. install.bat.txt

Issues

• nssm cannot it seems write stdout/err to the same file. More testing needed.

TODO

• test that we can supervise a couple of processes and kill them all properly in a python script wrapper. "Forking" doesn't exist in Windows, so we may hit some subprocess management issues.
• add windows-service.py script/entrypoint to job-runner, that parses .env file and launches both job-runner processes.
• add install.bat to job-runner, with some parameterisation for passwords and such, that install the above script as a service.

It occurs to me that it may be good time to see if we can consolidate into a single process, as it would save some pain here, perhaps. multiprocess module might be better than subprocess, as I think it has better x-platform support. Have we thought about just running the sync job in a thread? The only likely long blocking action I'm aware of is the docker copy, but we could launch that and poll for it finishing?

[sebbacon]

how will we install job-server? As pypi package or checkout?

I can't think a reason to strongly prefer one over the other, but just to note, when I installed a package on the L4, it ended up in a place that wasn't on my PATH.

For .env, as a team we've used https://pypi.org/project/python-dotenv/ before. Perhaps better to do it ourselves in plain python to remove dependencies though.

I defer to @evansd on running subprocesses.

[bloodearnest]

how will we install job-server? As pypi package or checkout?

I can't think a reason to strongly prefer one over the other, but just to note, when I installed a package on the L4, it ended up in a place that wasn't on my PATH.

So I think I have a slight preference for a checkout, because:

1) We have GH access - do we have pypi access? 2) It's much easier to edit the running code to add debugging or hot fixes

For .env, as a team we've used https://pypi.org/project/python-dotenv/ before. Perhaps better to do it ourselves in plain python to remove dependencies though.

Hmm, yeah, given it's not a requirement for local_run.

I defer to @evansd on running subprocesses.

[sebbacon]

We have GH access - do we have pypi access?

No we don't. In fact, I misread what you wrote, was thinking more pip install <github egg url thing> as the alternative to git pull

[bloodearnest]

Some decisions from a call with me and Dave:

1. We prefer a git checkout, as allows simplest update flow, and local debugging. We do need to provide requests and ruamel dependencies though. Investigating how best to do that via GH.

2. We will consolidate jobrunner into a single process, with the sync logic running in a thread. Some work needed on db connections and logging prefixes for this.

3. We will add a new module jobrunner.service or similar, which will handle parsing .env, and starting the sync and run main loops, and handle the keyboard interrupt from nssm.

[bloodearnest]

Update: the nssm config is installed on the server, and I was able to stop/start it as my regular user \o/

The nssm binary is /c/nssm-2.4/win64/nssm, I suggest adding this to your path. I will update the playbook shortly, and commit the batch script used to set it up into the repo also.

I've pulled down job-runner-dependencies into /e/job-runner/lib, and we've cowboyed a dummy simple service at /e/job-runner/jobrunner/service.py for now, which used sys.path hack to add ./lib.

TODO:

• add PYTHONPATH=lib environment variable to nssm config and get TPP to run it
• test it next week when it's quiet

[bloodearnest]

Update: some refinements, and some breakage

job-runner currently running as my user in a terminal, but natively on the host, and seems stable and anecdotally faster. . However, we accidentally broke the nssm setup. Fix is committed, waiting for TPP to apply it.

Permissions on /e/job-runner is possibly problematic, but have asked TPP to allow full control to level 2 users (or just devs) and the service user, with inheritance, which I think should mean everything works.

nssm now configured to run /e/job-runner/scripts/run.sh -m jobrunner.service, which simply sources the .env and execs python with it's args, so acts as a generic 'runner' for python, but with the right config.