Services

[grahamc]

@Profpatsch @shlevy and I had a meeting last week about managing development time services for lorri project environments.

Background

The story here is for a developer whose software has run-time service dependencies, possibly complicated ones. These services should start when the user is working on the project and should eventually shut down after the user stops work.

Services might be easy to "pre-package", (for example, but not necessarily using NixOS modules): mysql, postgres, kafka, zookeeper

Or might be custom to the user's organization, like another repo might be used to build software, and run that software as a service.

Or it might even be provided by the project itself, where the repo builds several executables, and lorri starts/restarts them.

Immediate Questions

How do we want to express service definitions? We should probably aim for simple to configure than very flexible .

Because a project might coexist on the same host as many other projects, do we want to ensure ports are assigned instead of requested?

Do we want to be able to run services remotely like using docker run?

Do we want to be able to run services in CI?

Long Term Implementation Ideas

• "Compiling" service definitions to launchd or systemd units, for use in the "user" agents. Requires expertise in how launchd works for implementation and users (for debugging and diagnosing issues.)
• Expressing the services using a subset of systemd's service definition format, and implementing a systemd-to-launchd converter (same trouble as above.)
• Use something like s6 (possible other inits?) to manage process starting. Ideally something with minimal bash scripts necessary.
• Build and launch services with Docker. Might work real nicely overall, though for services provided by the local repo, might be a bit awkward / slow.
• Support many copies of a given service. Ideally we can tightly limit the access a service has to the host, so their state can be carefully isolated, and erased.
  • Developers should be able to replace a local service with a remote service (ie: staging, production.)

Example application

The implementation of this feature should include an example application. Right now the thinking is a trivial "todo" manager written in Rust, backed by Redis.

Entering the directory and running lorri services start should start the services and make the webserver available on some local port.

Recompiling the code to the Rust application should cause the server to stop and start once the project has rebuilt. Because the application is hosted in the local repository, this restart should be triggered after the user runs cargo build.

Dogfood

Developers working on this project should use lorri services to manage their project's dependencies, and report bugs or fixes for UX problems. I'll be using it on OfBorg, as it depends on several external services, and several services built out of the local source.

---

cc https://github.com/target/lorri/issues/104

[jurpeedurp]

I like the idea of using something like https://www.nomadproject.io/ under the hood because the wheel wouldn't need to be reinvented and it seems to serve many of the short and long term goals stated while still being flexible enough to meet future use cases.

The following drivers are most likely of interest: https://www.nomadproject.io/docs/drivers/exec.html https://www.nomadproject.io/docs/drivers/docker.html

Jobs can be provided in JSON / HCL. See the spec here: https://www.nomadproject.io/docs/job-specification/index.html

[hanshoglund]

Nomad gives you the bonus of high-availability built-in, e.g. for shared/company-wide environments (it uses Raft under the hood).

[jurpeedurp]

Another benefit of this approach is that secret management is already a solved problem.

The nomad-vault integration could allow users to authenticate via token, username/password, kubernetes or against an existing backend / service and once authorized retrieve api keys, encryption keys, and other sensitive data that then can be injected into their environment.

[Profpatsch]

I like the idea of using something like https://www.nomadproject.io/ under the hood because the wheel wouldn't need to be reinvented

I agree that it’s not a good idea to reinvent the wheel, but I don’t know if it would be wise to tie ourselves to a specific solution that is itself tied to a company ecosystem (Hashicorp in this case).

From a quick skim of the documentation, Nomad seems to indeed cover a lot of the points we talked about, but also many, many more, which makes it hard to integrate into our small project. Maybe the right step here is to not implement anything at all, but provide some templates for users on how to integrate with frameworks that are popular at the moment (but might not be anymore in two years time).

But thanks for the links, please keep them coming! (especially if they describe the generic problem space and well-known (or proposed) solutions) Here’s the link collection issue: https://github.com/target/lorri/issues/104

[hanshoglund]

Maybe the right step here is to not implement anything at all, but provide some templates for users on how to integrate with frameworks that are popular at the moment (but might not be anymore in two years time).

In other words, coming up with service interface/abstraction that can be implemented in user space. Sounds reasonable idea to me, though I'm not sure what the interface points should be (CLI, Rust API, plugins?).

[Profpatsch]

Sounds reasonable idea to me, though I'm not sure what the interface points should be (CLI, Rust API, plugins?).

That’s a design space to be explored, I’d tentatively suggest something that can be described in either Nix or Rust, to “keep with the theme”.

Personally I’d start out with something fairly restricted (data format with well-defined semantics) and expand it or scrap parts based on user needs, until we have a good feeling of the scope we want to handle.

[jurpeedurp]

I agree that it’s not a good idea to reinvent the wheel, but I don’t know if it would be wise to tie ourselves to a specific solution that is itself tied to a company ecosystem (Hashicorp in this case).

From a quick skim of the documentation, Nomad seems to indeed cover a lot of the points we talked about, but also many, many more, which makes it hard to integrate into our small project. Maybe the right step here is to not implement anything at all, but provide some templates for users on how to integrate with frameworks that are popular at the moment (but might not be anymore in two years time).

But thanks for the links, please keep them coming! (especially if they describe the generic problem space and well-known (or proposed) solutions) Here’s the link collection issue: #104

I agree, that makes a lot of sense!

The one thought I'd like to add is that many, like myself, currently use nomad across multiple environments(production, staging, development). Maintaining the ability to describe tasks in a single format across environments is a huge win.

Without native nomad integration the idea that lorri requires a minimal amount of job-related configuration and allows execution of executables provided by nix/lorri(defined in shell.nix) in the current workspace would be incredible if possible.