Zookeeper based file config storage

[michalwarecki]

I'm using typesafe config in my application but we want to distribute the instances, provide synchronization and share configuration. We decided to use Zookeeper for that. Have you thought about implementation of Zookeeper based storage? Currently I see only file based storage. If you think this is a good idea, we can contribute that feature in a near future.

[havocp]

You could add a feature to the core for some kind of extension SPI that would let people add new sources and/or file formats to ConfigFactory.load. We talked about something similar for YAML support on another issue.

The core lib is zero dependencies and also I don't want to start collecting tons of backends I don't really use or understand, even if dependencies were OK. So I'd like to see an extension feature so that third parties can drop something like zookeeper support on the classpath and have it be picked up.

Hope that makes sense and happy to help talk over the extension mechanism. I think there's a lot of prior art out there such as slf4j, that's the sort of approach I have in mind. I'm pretty sure I've seen some blog posts or other write ups on best practices for Java SPIs. The main trick is that extensions have to be located by classpath search rather than explicit registration.

[JakubKahovec]

+1 for having support for a pluggable (etcd, zookeeper etc.) remote-based configuration. Something like https://github.com/rodrigosaito/spring-etcd

[havocp]

The next step here is for someone to investigate/propose/implement the needed SPI mechanism, so that new ways to load config can be added by dropping an extension jar on the classpath.

[dcowden]

+1, we need this too. In our case, we need an extension we can use to get and set values from Amazon KMS ( Key Management Services ) to manage credentials.

Outside of having an extension point, is the 'right' way to implement what we need to subsclass SimpleConfig and use it to decorate the standard, loaded config?

[havocp]

You shouldn't subclass Config (the docs point out that the interface will add methods), and SimpleConfig is private. Instead, create a Config (based on your KMS values) and then merge it in to other sources of config using withFallback. Best practice for libraries is to allow you to pass in your merged config, see https://github.com/typesafehub/config/tree/master/examples for how it should work.

[dcowden]

I understand but i have a problem that makes this approach difficult. Let me illustrate.

suppose i have this: myapp.db.password=DEFAULTPASSFORDEV

I can override something like this:

String passValue = kms.get("myapp.db.password"); Properties p = new Properties(); p.put("myapp.db.password", "passValue"); Config c = ConfigFactory.parseProperties ( p ) Config effective = c.withFallback(originalConfig);

But the problem here is that I need to enumerate all of the possible keys that could be used. This is not possible in KMS-- there's no api ( intentionally ) to just list all of the keys. You can only decrypt a value that you have stored. I had thought of doing something like this:

myapp.db.password=${kms:23920191019021}

When I see this type of key, i know to go to KMS, decrypt this, and substitute the password. In this way, i dont ever need to be able to enumerate all of the keys that i need to look up ahead of time.

So the question is, how can i inject an overridden value without having to know up front all of the values that could possibly by overridden?

[havocp]

300 would let you scan the config and analyze substitution nodes, perhaps. Don't know that there's a good way to do that right now.

[michalwarecki]

In next 2 weeks I shall write a sample spec and implementation of a SPI-based registry.

[havocp]

cool! feel free to bounce ideas off of us here or on the gitter chat.

[AnthonyClink]

Netflix has a great library (Archaius) for this based on apache commons. Supporting import of Apache Commons Configuration would be a great step in allowing developers to accomplish this.