User configurations

[veimox]

We use Ubuntu 18.04. We do config deployment through a debian package that installs package-X as dependency and right after our carbon configuration. We use a similar setup with many other packages and I would like to have the same setup here.

As of now, we have some "master" config files in the CONF_DIR that the user can override manually/via script. The problem is that we use the deb version that is deployed to all debian-based distros (check here) is loading the daemon with the configuration found in /etc/carbon/. Even though this is not related on how things are coded to this project, I think it gives a good context because many people will use the APT/YUM distributed version.

Because of that script, we are forced to use only the config files in there. For example the only way to change retention policies is editing /etc/carbon/storage-schemas.conf by hand or via script. Definitely I cannot do override that file with my own debian package because APT will complain (rightfully) that the file has been installed by another package.

In Debian based systems, this problem is tackled by having the XXX.d folders where there are optional/user configuration files that are imported. There order of preference is sequential, based on the name of the file which is usually preceded by numbers (e.g. 99-myconfig.conf). To see an example just run ls /etc/apt/apt.conf.d/ on a debian based system. This system works good for the deb-based config deployment I was describing at the beginning.

Summarizing, I would like to have an option of user configuration that extends/overrides a possibly default option as in the case of the APT version. A proposal could be to use the same UNIX like config method of having multiple config files that are included sequentially.

I wouldn't mind write the PR but first I would like to know your opinion :smile:

[piotr1212]

That works if you want to add entries to your config, for example you when you want to add a package repository. But how does that work if you want to replace the existing config? That file will be "owned" by the rpm/dpkg right? So you won't be able to replace it with another package.

I've seen people use packages for config management but they only did it for home built packages where they separated code from config, not for packages from the OS (which have application+config in one package). I think most companies are using config management tools like Ansible/Salt/Puppet nowadays or are using containers.

I'm not against your proposal, if you'll make a PR I'm happy to review/merge.

[piotr1212]

idea: add a --config option to the carbon binaries. change the service unit to use carbon-client.py --config /etc/mycarbon.conf, then you can have your config package own /etc/mycarbon.conf

[veimox]

Thanks for the fast reply! :D

Particularly for my case likely the only option will be to create my own binary. In such case, the easiest is what you propose of just changing the config folder. The reason is that even if we make a good solution here, will take too much time to get to upstream and likely I will not see it in ubuntu 18.04.

However, that doesnt stop us from finding a good long term solution that fit us all!

You've got a point there with the collision of multiple config files. If there some entry defined in the default file, and then redefined in a user config, it would expect that the user prevails. So user files will be overriding each other having the last the biggest priority.

Another use case would be extension of entries. Example: we have an entry X that defines a pattern and a retention. Then I redefine X in another file that only redefines the retention. Should it fail because is miss configured or just override the retention of the first entry?

I personally dont like the idea of extension because is more error prone and hides info just to save up one or two lines.

On Fri, May 15, 2020, 20:13 Piotr Popieluch notifications@github.com wrote:

idea: add a --config option to the carbon binaries. change the service unit to use carbon-client.py --config /etc/mycarbon.conf, then you can have your config package own /etc/mycarbon.conf

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/graphite-project/carbon/issues/895#issuecomment-629405641, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABCCAY4YV6YRNQW4VTAOZKDRRWA6FANCNFSM4NBURVGA .

[piotr1212]

The following config directories make sense to me: carbon/storage.schemas.d carbon/storage.aggregations.d carbon/whitelist.d carbon/blacklist.d carbon/aggregationrules.d carbon/relay.rules.d carbon/rewrite.rules.d

For carbon.conf splitting it doens't make much sense.

wrt your problem. I now remember how they solved this. They had a -config package install a file in /etc/deamonxx.conf.something and have the post-script copy it over to /etc/deamonxx.conf. But really, just use one of the many available config mgmt tools instead of building your own.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.