package config files for query and collector service into the jar

[codefromthecrypt]

The query and collector service include pre-canned scala config files. These should be packaged into the jar and read from the classpath.

[dsyer]

The consensus seems to be that config files should not be scala as well. In the meantime, though, can we just put them in the jars so at least they are self contained (and can be used if exploded)?

[eirslett]

How should one override the configuration then? I think they should be kept outside of the jar until we have replaced them with something else. We need at least some kind of support for runtime configuration? E.g. the Cassandra host...

[dsyer]

I think we need something that works out of the box on localhost at least. I take the point though, the config files we have are of limited use as they are. Flags seem to be the way to go.

[eirslett]

With some form of hierarchy?

-backend=cassandra -backend-cassandra-host=123 -backend-cassandra-port=456

-backend=X -backend-X-Y=...

We also discussed using Guice, and letting users wire up the dependencies as JSON.

[dsyer]

Spring Boot would also provide a nice abstraction for externalized configuration (properties, YAML, command line, system properties, env vars). You wouldn't need a huge raft of dependencies just to bind to the environment - probably just a couple of extra jars. And it has support for executable jars in a nicer format than the shadow ones.

[eirslett]

spring-core maybe, and xml-based configuration? The Finagle stack doesn't use servlets, only pure Finagle HTTP.

[eirslett]

But this issue is derailing into a duplicate of #466

[dsyer]

Indeed, the wider discussion about configuration should be in #466. How about limiting this one to just packaging the existing scala config files until we have something better? They won't weight down the jar file.

[eirslett]

So we put them in src/main/resources or something?

[dsyer]

Or just add config to sourceSets.main.resources (or whatever the gradle incantation is). Might be a good idea to map it to a subdirectory in the jar as well (so maybe use the jar configuration instead).

[dsyer]

Actually src/main/resources makes some sense as well - just means that the bin scripts have to be changed again.

[eirslett]

Your point about bin scripbs is quite good; changing them isn't trivial (given some people may have put the bin scripts in their puppet setup etc)

[codefromthecrypt]

I think there are a couple issues going on here.

One is that we have a few pre-canned configurations inc dev, cassandra, redis. These should be able to run self-contained and be configured with flags.

This is an essential part of our quick start story. The 'pre-canning' can happen inside the fat jar which is an alternative to the old dist thing (as it doesn't use bin scripts). In other words, fat jar becomes self-contained and the quick start story itself!

The fat jar could also set its classpath to include cwd first. This would allow someone to use an external path to override the old school Scala loading files until we have a better solution.. (#466) which we will need as the current approach is deprecated and also no longer used at Twitter.

Wdyt?

[jamescway]

+1 flags. We've had success using and adding flags which have 1) default values, 2) prints usage if misused, 3) in the compiled code, 4) already being used in many places. Only con was the documentation was hard to find.

[codefromthecrypt]

ok cool things (including flags) -> #466

This one will solve the immediate issue, which is make it possible to use packaged configs

[codefromthecrypt]

@dsyer @spencergibb PTAL at #540