search

weavejester / integrant

Micro-framework for data-driven architecture
MIT License
1.24k stars 64 forks source link

Multiple keys with same init and halt! behavior #9

Closed dsteurer closed 7 years ago

dsteurer commented 7 years ago

It might be useful to have an official way for sharing init and halt! behavior among keys. One option is to use derive and the hierarchy system for multimethods. However my impression is that this feature is only rarely used in general and that it might not be the right way to solve this particular issue. In general it seems useful to be able to specify the init and halt! behavior for a key as part of the config as opposed to somewhere in the code. (I think Duct is doing something similar.) For example, if the value for a key is

{:methods 
 {:init foo.component/init
  :halt! foo.component/halt!}
 :options
 {,,,}}

the init behavior could be to apply foo.component/init to the (expanded) options, and add foo.component/halt! as meta-data (similar to the current ::build meta-data). The halt! behavior would be to apply foo.component/halt!. With this implementation, it would be easy to share init and halt! behavior (one could even use Refs). This implementation would also support a more concise form

{:methods foo.component/methods
 :options {,,,}}

where the value of foo.component/methods has the appropriate shape.

Could it make sense to integrate this kind of implementation with the current code base?
One option is to change the :default method for init-key such that it checks if the value for the key has this form (and maybe also if it has some metadata tag like ^:custom-methods or so). A much more drastic change would be to just replace the current multimethods with this kind of implementation. The main downside I can see here is that it is not clear how to achieve the same level of conciseness as with multimethods. However, a potential advantage is that it seems to solve #8 and maybe also #6.

weavejester commented 7 years ago

The official way is to use derive. Multiple inheritance is a specialised tool, but perfect for this use case.

Let me explain a little about my reasoning, and where I'm going with Integrant.

I want to make configurations discoverable, so in most cases you can programmatically ask, "What's the web server?" or, "What's the database connection?". This opens up ways of querying and transforming configurations in intelligent ways.

In order to do this, I think we need to encourage people to use a uniform language. Namespaced keywords are useful in this regard, because we can use them to produce taxonomies of meaning. For example:

(derive :duct.server/jetty :duct.server/http)

If we want to find the web server of a configuration, we search for a key that's derived from :duct.server/http.

What I want to avoid is forms of configuration that avoid semantic meaning. In your example:

{:methods 
 {:init foo.component/init
  :halt! foo.component/halt!}
 :options
 {,,,}}

What do we know about this component? Only how to initiate and halt it. We have no information about what it does, or what properties it might have. On the other hand, consider a configuration like:

{:foo.component/bar {,,,}}

What do we know about this? Well, we know the key name and namespace, and if we push the configuration through the load-namespaces function, we can attempt to load the namespaces associated with the key. In this case, foo.component and foo.component.bar.

(It seemed like a good idea to allow keys like duct.server/jetty to try to load both duct.server and duct.server.jetty rather than have to resort to keys like duct.server.jetty/jetty)

So maybe we then discover a namespace foo.component.bar that looks like:

(ns foo.component.bar
  (:require [integrant.core :as ig]))

(derive :foo.component/bar :duct.background/worker)

(defmethod ig/init-key :foo.component/bar [_ config]
  ,,,)

(defmethod ig/halt-key! :foo.component/bar [_ process]
  ,,,)

Now we know how to initiate and halt the component, but we also know that this key acts semantically like a :duct.background/worker. Maybe we know this means that the component has a :workers and :queue key.

Is my reasoning behind the design a little clearer, now?

dsteurer commented 7 years ago

I agree that attaching semantic meaning to the keys is elegant. However, I am not sure I see much difference in terms of conveying semantic meaning and being discoverable between the following two options

{:duct.server/jetty 
 {,,,}}

and 

{:duct.server/http
 {:methods duct.server.jetty/methods
  :options {,,,}}}

I was suggesting the second form because it makes it very easy to declare multiple components with the same init and halt! behavior in the config, e.g., two http servers with different port and handlers (but maybe using the same database). To achieve the same with derive, I would do something like 

(derive :duct.server/jetty-1 :duct.server/jetty)
(derive :duct.server/jetty-2 :duct.server/jetty)

and have :duct.server/jetty-1 and :duct.server/jetty-2 as keys in the config. Then the isa? based dispatch for multimethods will use the :duct.server/jetty methods for both components. This approach seems fine but I am not sure where the derive code belongs. It seems to me that it should be part of the config. It certainly doesn't belong to the duct.server.jetty code.

I think I want to use derive here in a different way than in your examples. In your examples, you did not seem to intend to use the hierarchy for isa? based dispatch of the init-key and halt-key! multimethods. Instead you were using it to declare some kind of contract satisfied by the component (somewhat reminiscent of protocols).

I guess other options for the config format that interpolate between the two above is something like

{:duct.server/http [:duct.server.jetty {,,,}]}

and have the multimethods dispatch based on the first element of the value (as opposed to the key) or 

{:duct.server/http ^:duct.server/jetty {,,,}}

and (optionally) use the metadata to dispatch .

In general, my feeling is that the current implementation uses keys for two distinct purposes: 1.) describe the dependency / reference structure of the system 2.) select the init and halt! behavior for the components

I think it would be useful to have a mechanism to resolve conflicts between those two purposes.

weavejester commented 7 years ago

While it's possible to add in symbols to the config that reference functions directly, I don't want to tie the configuration to a concrete implementation. I want a clear dividing line between configuration and implementation, in much the same way that tagged data in edn is separate from the data readers that interpret them.

Regarding how you'd implement two servers, you'd just create a new namespace for them. For example, if your project name was foo:

(ns foo.server
  (:require duct.server.jetty))

(derive ::web-server :duct.server/jetty)
(derive ::api-server :duct.server/jetty)

And in your configuration:

{:foo.server/web-server {,,,}
 :foo.server/api-server {,,,}}

In practice, I don't expect this will be necessary often, and should be discouraged in general because it introduces complexity.

For example, if I have one web server, I can easily find the entry point of the application. If I have multiple web servers, the entry point is no longer obvious.

Integrant is designed to shepherd users toward configurations that are simple and semantically descriptive. It's still possible to have multiple web servers or database connections, but it needs to be a deliberate choice.

dsteurer commented 7 years ago

These are good points. Thank you for the explanation.