Daemon configuration via environment variables or CLI arguments

[sudoforge]

Description

Currently, the daemon reads settings/options for the server from murmur.ini. This is a fine solution for most cases, and works for most users, however, there are various projects, like my own docker-murmur, that would benefit from enabling envvar or cli-based server configuration. Right now, in order to configure the server, users of my project need to either:

• Provide a pre-written murmur.ini file, or

• Set values in environment variables, which I then use to replace the default values in the default murmur.ini.

While this works, I think we could do better (and enable me to ditch dependencies on sed and other tooling!)

Proposal

Enable setting configuration via environment variables or command-line arguments. This will allow projects like mine to continue to enable our users to easily configure the murmur daemon, while ditching CLI tool dependencies required to map to the murmur.ini file.

What this might look like

via command-line arguments...

$ /opt/murmur/murmur.x86 -fg -o serverpassword 'myPass' -o defaultchannel 'some channel name'

via environment variables...

$ MURMUR_SERVERPASSWORD="myPass"
$ MURMUR_DEFAULTCHANNEL="some channel name"
$ /opt/murmur/murmur.x86 -fg --use-envvar-configuration

Potential roadblocks

Any potential roadblocks discovered through discussion will be listed here.

---

Please feel free to discuss your opinions and ideas regarding this change.

[mkrautz]

You shouldn't really need to use sed to replace the default values of murmur.ini. Why not just...

echo > murmur.ini.tmp  # clean file
if [ "${MURMUR_BANDWIDTH}" ]; then
    echo "bandwidth=${MURMUR_BANDWIDTH}" >> murmur.ini.tmp
fi
...
mv murmur.ini.tmp murmur.ini

? I.e., create the whole file from scratch?

I believe we have sane defaults for almost all settings. If not, we should change that.

I'm not entirely opposed to the idea, but I think that this is quite easily implementable outside Murmur in a small script.

Alternatively, we could add some magic to the .ini reader to allow for variables. So, for example:

Env var variables:

defaultchannel=${env:MURMUR_DEFAULT_CHANNEL}
bandwidth=${env:MURMUR_BANDWIDTH}

General variables:

myname=MyServer
registerName=${myname}
welcometext="Welcome to ${myname}! Enjoy your stay!<br />"

I'd prefer that to having env vars for all options. The downside of that would be that there is no way to fall back to the defaults -- the defaults will have to be in env vars.

Also, please note that the defaults in murmur.ini are just that -- defaults. They are typically overwritten via per-vserver config options set via RPC (DBus/Ice/GRPC)... In a typical scenario, a user would use the default murmur.ini, and set any custom settings via RPC.

[mkrautz]

Oh, and I'd like to add that I'm also not strongly opposed to adding a CLI flag, as you suggest. But I'd worry that people accidently leak secrets that way.

[sudoforge]

You shouldn't really need to use sed to replace the default values of murmur.ini. Why not just...

echo > murmur.ini.tmp  # clean file
if [ "${MURMUR_BANDWIDTH}" ]; then
    echo "bandwidth=${MURMUR_BANDWIDTH}" >> murmur.ini.tmp
fi
...
mv murmur.ini.tmp murmur.ini

This still leaves me with some cli tool dependencies, which is what I'd like to be able to move away from, so that I can run from scratch.

FROM scratch is basically a no-op in docker. It's intended to be used for super-minimal images that run a binary, for example murmurd. This is currently possible if I add the binary from the releases and copy a murmur.ini into the image. This isn't ideal, though, as one of the unique selling points of my project is that instances can be started and configured entirely by environment variables, which the user sets on the command line, e.g.:

docker run -e MURMUR_DEFAULT_CHANNEL='my channel' -e MURMUR_BANDWIDTH='whatever' bddenhartog/docker-murmur

This makes it stupidly simple to quickly start lots of instances with different configurations.

Alternatively, we could add some magic to the .ini reader to allow for variables. So, for example:

Env var variables:

defaultchannel=${env:MURMUR_DEFAULT_CHANNEL}
bandwidth=${env:MURMUR_BANDWIDTH}

I could totally get behind this. This is an extremely nice QOL improvement, in my opinion.

I can understand your concern with leakage if a CLI flag was exposed. Perhaps this shouldn't be included.

[SISheogorath]

From other projects I can say that the regular way to implement environment variables as options for configuration has the following order:

environmentment variables > values in config file > hard-coded defaults

Means that environment variables overwrite all settings while hard-coded defaults are used as "worst case".

Common way is to prefix environment variables to avoid collisions. So as we do in the docker image something like MURMUR_.

For secrets there should be a command line option. But maybe not to pass the secret directly but to pass a path where the application can read it. Background: Docker started secret management in newest stable version 1.13.0. But for non-swarm-mode usage they should maybe read from environment variables, too. That's not very nice but needed.

Addition: I would really prefer a complete integration of the environment variables into murmur directly instead of using the config parser. The reason is that this still results in a duplication of code in speech of the customized murmur.ini.

_{No offence}

[mkrautz]

The reason I'm wary of implementing this is that it's somewhat of a half-assed solution.

Not all settings can be configured via the .ini, so while enabling all the .ini facing settings to potentially be configured via env vars, your users would still be stuck with RPC for further modifications. And if they need RPC for that, why not use it when provisioning the initial server in the first place?

Also, after first run, the original defaults from the .ini might have been persisted into the per-vserver config in Murmur's SQLite file. This means that changing the .ini will have no effect on that server. Only RPC changes will. (I am not sure this is relevant for the Docker image -- I don't know if updating the settings and re-deploying is a valid use-case?)

Yes, this is mostly a Murmur issue. Server configuration is not very intuitive -- and not having an easy-to-use CLI configuration tool that gives you the same power as the RPC tool is simply too bad -- but it's where we are today.

I realize the deployment scenario you're looking at with murmur-docker is probably only to have a single vserver per deployment, which I can appreciate. If that is the only use-case, then I agree -- the feature request makes sense. But if docker-murmur is also intended to be used with multiple vservers per instance, I don't think this proposal will suffice. Can you comment on that?

To move the discussion along, I would like to know how many settings you're able to set via environment variables in docker-murmur currently? And how is that system working out for you? Is it even sufficient for configuring Murmur?

This feature request is not something that's high on our priority list, so for everyone's sake I would really recommend doing this as an external mechanism that generates a mumur.ini based on the current environment variables. That way, we can get some real world experience with the feature. If it's popular, and makes sense, we can simply implement support for the same env vars in Murmur itself.

[sudoforge]

But if docker-murmur is also intended to be used with multiple vservers per instance, I don't think this proposal will suffice. Can you comment on that?

You're correct in assuming that docker-murmur isn't intended to be used with multiple vservers per instance; instead, it's on a 1:1 ratio of instance to vserver. If a user wanted to run multiple vservers in a single instance of docker-murmur, I suppose they could (we allow for setting up via ICE), but I haven't had a need to do this and am unsure if the existing workflow would need to be changed at all (I don't think so, but then I may be missing something that isn't coming to mind at the moment).

To move the discussion along, I would like to know how many settings you're able to set via environment variables in docker-murmur currently? And how is that system working out for you? Is it even sufficient for configuring Murmur?

Current configuration options can be seen here. Last time I audited, we supported 100% of the settings that made sense for a containerized environment. We omitted, for example, options like dbus configuration (as it is deprecated), as well as options like port and host as they are unnecessary in our environment. Things like custom SSL certs are uploaded by the user and stored on the host machine, which are then mapped into the container.

Currently, I use an orchestration system to manage initializing instances. An end user specifies their desired configuration in the front end, which is sent to an endpoint that ends up starting a container (setting the required settings via our setup script, using environment variables whose values are based on the user's supplied params) and returns the public connection information to the end user. I have a (pretty small) collection of instances, mostly for people I know directly, and some of their acquaintances. Nothing formal, and this setup works well for that. I don't have access to usage information from the people who use the docker-murmur project, which at last count had been pulled a fair amount (a little over 15k -- nothing huge, but I don't think there's only one person using it, is what I'm getting at).

In truth, all this feature request would really do is allow us to minimize the size of our docker-murmur image, which isn't that large to begin with, so it's definitely not something I would categorize as a high priority either -- although it would be a nice QOL improvement (specifically, for the docker-murmur project; not sure how/if these changes would benefit others).

Psst - sorry for the delay in replying, I was out of town!

[mkrautz]

We'll probably add the ability to configure Murmur via command line args and/or environment variables sometime post 1.3.0.

It would also be nice to have for a Murmur test-suite.

[sudoforge]

We'll probably add the ability to configure Murmur via command line args and/or environment variables sometime post 1.3.0.

Exciting news -- that's great to hear!

[sudoforge]

@mkrautz Not sure if there's a better forum for this, but I'd like to open a dialog with you and/or the rest of the Mumble team about submitting the docker image I maintain to the Docker team as the official Murmur image. There are a few changes that @SISheogorath and I have talked about that we'll need/want to make before submitting the image to the official library, but they're mostly cosmetic things, like making the documentation a bit more new-user-friendly. I have a few questions for you regarding version support and release cycle, but before we get into that, is this something you think the team would be open to?

It should be noted that official images don't need to be managed/released/approved by the project's team -- I plan to maintain responsibility for the docker image, unless your team would like to take over the project and integrate it into core. And although it's not necessary to gain you/your team's approval, I'd definitely like to do that before submitting the image to the official Docker library as it will have your project's name on it.

[fgaz]

I ran into the same problem while trying to use nixops' key management. There seems to be no way to obtain the password externally (either from an env var or a file) :-/

[Avatat]

@Kissaki, @mkrautz, @Krzmbrzl, can I try to solve it? I'm more a SysOp than a developer, but I would have a try :)

[Krzmbrzl]

@Avatat absolutely! :)

If you create a WIP PR early on, we can directly start helping out and commenting on the code if this is needed :+1:

[Avatat]

Wow, that was fast!

I'm cloning the repo and starting looking for the configuration's parts of code.

[Krzmbrzl]

If you have any questions about that feel free to ask them on our IRC channel (or here if you prefer that) :point_up: