[RFC] Optional components in a system

[SevereOverfl0w]

Problem

There are a few cases in which you want to swap out entirely different components into your system. For example, switching development stubs for real production components, or only running kick during development and using one-shot builds in production.

Status

I'm looking to gather feedback on the below solutions, and possibly any additional solutions that can be considered.

Proposed Solutions

:component/remove?

Pros

• Clear what is going to happen.
• Clear use with all aero conditionals

Cons

• Requires enumeration of all NOT cases in conditionals. This is not easy with, e.g. #profile. For example, to only have a component in production:

  :foo {:component/remove? #profile {:dev true :test true}}

• Production components with dependencies are unable to be grouped with their dependencies as part of the conditional. This makes it harder to mentally follow code paths as a group.

:component/keep?

If (and (contains? x :component/keep?) (not (:component/keep? x))) then remove the component.

Pros

• Clear what is going to happen (but less clear than :component/remove)
• Clear use with aero conditionals

Cons

• As a contains? check is in place, this violates some expectations of the default value for this key being nil (falsey)
• Production components with dependencies are unable to be grouped with their dependencies as part of the conditional. This makes it harder to mentally follow code paths as a group.

Subsystem

There are a few ways this could look, and each of them have more/less obvious syntax to them. I think the below is the most obvious syntax that's valid edn.

:ig/system
{:web-server
 {:emailer #ig/ref :emailer}

 #subsystem :email-subsystem
 #profile {:dev {:email-stub 5}
           :prod {:smtp-emailer {:smtp-conn #ig/ref :smtp-conn}
                  :smtp-conn {:host "127.0.0.1" :username "user"}}}

 ;; nil value (as would be for the :production case) results in no alteration.
 #subsystem :dev-util
 #profile {:dev {:some-dev-util "blah"}}
}

The result of this in production is a system like so:

:web-server {:emailer #ig/ref :emailer}
:smtp-emailer {…}
:smtp-conn {:host "…" …}

That is, there is no namespacing from the nested subsystem.

Pros

• Groups branching points and their dependencies together
• Unspecified conditional branches are removed automatically

Cons

• Introduces new syntax
• Naming a subsystem may imply namespacing, but implicit namespacing would break #ig/ref
• I can imagine uncertainty over usage of #ig/ref inside a subsystem/across subsystem boundaries.

#merge

Encourage #merge for this case:

:ig/system #merge [
  {:web-server {:emailer #ig/ref :emailer}}
  #profile {
   :prod {:smtp-emailer {…}
          :smtp-conn {…}}
   :dev {:emailer-stub {}}
  }
  #profile {:dev {:some-dev-util "blah"}}
]

Pros

• Uses existing features of aero
• Builds on clojure semantics

Cons

• Requires converting existing call of a {} to a #merge []
• Requires breaking up system into lots of smaller systems in order to have some key proximity with regards to logical grouping of components (e.g. putting email related components near things that use email things inside the map).
• Looks a bit ugly. This is personal aesthetic though, not a strong argument.

Implicit merge

Instead of only considering the :ig/system key, a :edge.system/overlays key could be considered also. This would be a map of named subsystems which would be merged in. Conflicting keys in maps would be an error.

{:ig/system {:web-server #ig/ref :emailer}
 :edge.system/overlays
 {:emailer-impl #profile {:prod {:smtp-emailer {…} :smtp-conn {…}}
                          :dev {:emailer-stub {}}}

  :dev-utils #profile {:dev {…}}}}

Pros

• Low syntax
• Clear use with aero built-in conditionals
• Named branching points allows for conflict detection with useful information, e.g.

  :dev-utils and :emailer both contain :smtp-emailer, and none has priority. :boom:

• Allows grouping dependencies of branching points together
• Unspecified conditional branches are removed automatically

Cons

• Implicit merging means that it is more "magical". Good naming & logging will help reduce the impact of this.
• Naming a subsystem may imply namespacing, but implicit namespacing would break #ig/ref

[malcolmsparks]

Useful also for feature toggles. I think component/keep?is the more obvious, or even component/when. Might @weavejester have an opinion because this feels like it belongs in Integrant.

[weavejester]

The way I'm currently handling this in Integrant is to leave it to a secondary layer. For example, Duct has "profiles" for this situation.

[danielcompton]

I’ve found for coarse grained distinctions between dev/prod/test that it is easy to just have three component constructor functions for the different contexts. Just my 5c.

[SevereOverfl0w]

More detail on what Duct does: https://github.com/duct-framework/duct/wiki/Configuration

A second system is merged in. This could also be represented in edge by a second key in the config.edn.

---

@danielcompton how do you handle dependencies of those components? For example in the emailer example above. Does this also imply that your constructors are always configured the same way?

[RickMoynihan]

In a few of our systems we use duct profiles for this and they work quite well. We have quite a complex multi-tennant app with lots of feature toggles; and have a cascade of profiles which include a base profile, a customer specific layer, and then various others for env (prod/dev/etc) local overrides etc.

Some other non-duct systems use a variety of approaches including #merge manual merges, different entry points etc.

Also worth saying you can provide different sets of keys to integrant to start which can be used for optional systems that don't share explicit dependencies.

[SevereOverfl0w]

Also worth saying you can provide different sets of keys to integrant to start which can be used for optional systems that don't share explicit dependencies.

I was made aware of this recently. I'm undecided as to it's general usefulness :smile:. Seems quite easy to get yourself into a bit of pickle in explaining about "entry" components and dependent components, and whether or not they need adding to certain lists.

In a few of our systems we use duct profiles for this and they work quite well. We have quite a complex multi-tennant app with lots of feature toggles; and have a cascade of profiles which include a base profile, a customer specific layer, and then various others for env (prod/dev/etc) local overrides etc.

This is really useful feedback @RickMoynihan, thanks!

A similar option here might be that the implicit merge approach is a map like so:

:ig/system {}
;; TODO: Name?
:ig/overlays {
  ;; Approach Structure 1
  :profile #profile {
    :dev {}
    :prod {}
  }
  :user #user {
    "dominic" #include "/home/dominic/system.edn"
  }

  ;; Approach Structure 2
  :emailer #profile {…}
  :other-subsystem #user {
    …
  }
}

This allows you to get far more complicated with your conditionals, and forces you to name your branching points.

I think that both the implicit merge of some kind of "overlay", and :component/keep? are strong candidates with this extension in place. The preference for either likely hinges on the kind of conditionals you do.

Will update OP.

[RickMoynihan]

This looks very much like you're just rebuilding duct profiles.

Personally I'd like to see the module system bits of duct fully extracted from the web framework so it can be used elsewhere (it's almost been this since the start, and I wouldn't be surprised if this has actually happened, or will happen soon).

:component/keep?/:component/remove? looks to me like ^:displace/^:replace which you can use in duct meta-merge's.

Also worth saying you can provide different sets of keys to integrant to start which can be used for optional systems that don't share explicit dependencies.

I was made aware of this recently. I'm undecided as to it's general usefulness 😄. Seems quite easy to get yourself into a bit of pickle in explaining about "entry" components and dependent components, and whether or not they need adding to certain lists.

I certainly find it useful. One place I use it is to start subsets of the system for unit/integration testing; whilst using the production/base config (sometimes with a few test profile overrides merged over the top). Another place it is useful is for running an app in different modes without having to write lots of complex command line argument code. e.g. I can ship an uberjar for an application which takes an sequence of keys as a command line argument. If the keys aren't provided we just start all derivations of :duct/daemon; if it is provided we start those specific components. One of those components could for example create your SQL tables if you provide the key :app.schema/create so users can initialise the application before they run it.

Without this you need to either explicit add custom mains which is fine, but then they'll want to use the system anyway, so you'll find each -main entry point is largely duplicated and risks becoming subtly inconsistent in style. Or you'll end up always requiring the schema initialisation code in a single common main entry point, or just maintaining that list of keys in the app, and parsing the command line args to decide when to start the system.

All things being equal there's not a lot of reason to prefer one approach over another; as they're essentially equivalent.

[hammonba]

I'm not a fan of add/remove tags; I think they just add complexity when building the system (Add this component UNLESS there's an 'r' in the month OR my hair is green today).

I prefer to compose the system that I do want out of smaller reusable parts, which makes a vote for #merge.

I also like the idea of using integrant composite keys as marker interfaces and starting refsets; it makes it obvious which bit is started on each profile.

[hammonba]

for project kermit we defined a new trivial integrant component like so

(defmethod ig/init-key :edge/vicarious
           [k v] v)

and then wired it in to integrant subsystems like this

 :ig.system/standalone
 {
  :kermit.nordics.local-filebucket/create
  {:kermit.email-service/stub {:log-stream #ig/ref :kermit.riemann/logstream}

  [:kermit.communications/email-service :edge/vicarious]
  #ig/ref :kermit.email-service/stub
...

 :ig.system.connected
{
  :kermit.email-service/sendgrid
  {:config #ref [:email.config/sendgrid]
   :log-stream #ig/ref :kermit.riemann/logstream}

  [:kermit.communications/email-service :edge/vicarious]
  #ig/ref :kermit.email-service/sendgrid
...

and we can then use #ig/ref [:kermit.communications/email-service :edge/vicarious] from the rest of the system when we need the email component

and then we choose between those two subsystems at the top level

 :ig.system.profile/connected #merge [#ref [:ig.system/connected]
                                      #ref [:ig.system.nordics/base]]

 :ig.system.profile/standalone #merge [#ref [:ig.system/standalone]
                                       #ref [:ig.system.nordics/base]]

 :ig/system
 #profile {:prod #ref [:ig.system.profile/connected]
           :staging #ref [:ig.system.profile/connected]
           :dev #ref [:ig.system.profile/standalone]}

this keeps the the configuration intentional and (hopefully) unsurprising.

I would describe this as equivalent to a symbolic link, but for integrant configuration

Hmmm perhaps :edge/sym-link would be a clearer term than :edge/vicarious

[RickMoynihan]

I've used integrant with aero in the past, on projects which couldn't easily be ported to duct. It works fine and I've essentially done things like what @hammonba says.

Duct profiles (meta-merge), are less expressive than aero, but more predictable and less powerful. I'm a big fan of aero but it's essentially turing complete when you consider that you have various ways of expressing conditionals, and #includes could in principle at least build loops. Incidentally I suspect duct is technically turing complete too, but feels at least a little more declarative.

The bigger issue with aero + integrant is that it can quite easily become an unstructured mess of references/merges etc. However, I've done it myself, and it's not that bad. On the flip side also duct profiles sometimes feel like they're not quite powerful enough, which can sometimes lead to config being duplicated. For example today we were trying to conditionally toggle feature-v1 and feature-v2, where the features were effectively a profile; so we ended up writing our own conditional merge tagged literal. (We could've used aero, but want to avoid the expressivity).

[SevereOverfl0w]

There is a limitation to this approach that we have run into on projects. Assume you have a tag called #aws-ssm which utilizes Aero's delay functionality so it doesn't trigger outside of production. If you have a top-level key like "connected" in the above example which does like so:

:ig.system/connected
{:email-sender
 {:secret-key #aws-ssm "secure-string-key"}}

It will blow up. So you have to create your integrant layers quite carefully:

:ig.system/email-system
#profile {:dev {:email-stub nil}
          :prod {:email-sender {:secret-key #aws-ssm "secure-string-key"}}}

A guard would also prevent exceptions, but I don't think that improves readability:

:ig.system/connected
#profile {:prod {:email-sender …}}

---

I like the way this reads:

:ig/system #profile {:dev #merge [#ref [:ig.system/base]
                                  #ref [:ig.system/email-system]]
                     :prod #ref [:ig.system/base]}

I think I would recommend this over:

:ig/system #merge [
  {:web-server …
   :routes …}
  #profile {:dev 
            {:email-stub nil}
            :prod
            {:email-sender {}}}
]

The latter form is harder to read and as your system grows.