Make profiles resolve recursively?

iku000888 commented 5 years ago

Old behavior of `:duct.core/include` key

config foo.edn

{:foo/foo 1}

config bar.edn

{:bar/:bar 2
 :duct.core/include ["foo.edn"]}

config.edn that includes bar.edn

{:duct.core/include ["bar.edn"]}

And prep'ing config.edn would result in

{:foo/foo 1
 :bar/bar 2}

Trying something similar with `#duct/include`

#duct/include assumes the expanded config map will be under a profile key, so simple replacing :duct.core/include with #duct/include will not do.

e.g.

config foo.edn

{:foo/foo 1}

config bar.edn

{:bar/bar 2
 #duct/include "foo.edn"}

config.edn that includes bar.edn

{:some/profile #duct/include ["bar.edn"]}

does not recursively resolve.

Some more contexts from slack convo

selection_146

selection_147

weavejester commented 5 years ago

Thanks. Could you elaborate on your use-case a little further? How exactly are you splitting up your configuration, and why?

iku000888 commented 5 years ago

The one I had is a datomic-connection component that is used by a kafka-streams component that is used by the app.

The datomic-connection configuration has to be provided by the app as during development I would use datomic-peer and in production I would use datomic cloud which we would switch via profiles.

Some more reasons doing this style were that we have several micro services of this manner and config files were getting really huge so it was a good way to split things up in a hierarchical manner.

The app is something like this;

App repository

Both files have the config for :datomic/connection

datomic-peer/datomic-connection.edn (in the classpath when datomic-peer profile)
datomic-cloud/datomic-connection.edn (in the classpath when datomic-cloud profile)

config.edn

{:duct.core/include ["stream-processor.edn"]}

Kafka streams lib repository

stream-processor.edn

{:duct.core/include ["datomic-connection.edn"]
 :kafka-stream-component/stream {:datomic-connection #ig/ref :datomic/connection
                                                           ;; and other configs....
                                                           }
 }

weavejester commented 5 years ago

While this isn't a general solution, if you're just choosing the values of a single key, then you could write:

{:datomic/connection #duct/include "datomic-connection.edn"
 :kafka-stream-component/stream {:datomic-conneciton #ig/ref :datomic/connection}}

And then in datomic-connection.edn:

{:uri "datomic://..."}

Or whatever configuration you want.

danielytics commented 5 years ago

I found this issue while looking for a similar solution, so let me explain my use case:

I wanted to include migrations in a separate config file, both the duct.migrator.ragtime :migrators list and the migration components themselves, as the main config file is getting large and I feel that migrations should live in a separate file. I can of course include the :migrators list easily, but how do I also merge all of the individual migration components into the base profile? The old :duct.core/include would have done the trick, but I'm unsure how to solve it with just the reader literal.

I was thinking maybe I could create a migrations profile and include the migration components into that, but I don't understand how to use profiles other than the defaults. Can I make that profile always be merged in (regardless of dev or prod) and is there any documentation for using custom profiles? I didn't find any.

weavejester commented 5 years ago

I was thinking maybe I could create a migrations profile and include the migration components into that, but I don't understand how to use profiles other than the defaults. Can I make that profile always be merged in (regardless of dev or prod) and is there any documentation for using custom profiles?

There needs to be more documentation on this, I agree! If you take a look at your current configuration, you'll see that there are three profiles:

:duct.profile/base - which is always included
:duct.profile/prod - included only in production
:duct.profile/dev - included only in development

So what do you do if you want to create your own profile? Just inherit from the one whose behavior you want in your config.edn file:

{[:foo.profile/migrations :duct.profile/base] #duct/include "foo/migrations"
 ...}

If you want your config file to be less cluttered, you can move the inheritance to src/duct_hierarchy.edn:

{:foo.profile/migrations [:duct.profile/base]}

And then in your config.edn you can just write:

{:foo.profile/migrations #duct/include "foo/migrations"
 ...}

danielytics commented 5 years ago

Thank you very much, that's perfect!! Thanks for a great library, by the way.

duct-framework / core