visualizing dependency graph

[yogthos]

it would be a nice feature to be able to print out the dependency graph that ends up being generated based on the namespaces

[tolitius]

that's a great idea.

Yesterday at Datomic conf (pre clojure/conj), I talked to @stuarthalloway about application state in general, and he mentioned that visualizing a state of an application with all the dependencies and their state would be really valuable.

I did mention before that mount does not have it at the moment, and there is something that needs more thinking:

Currently a state does not really know it depends on another state, it just has start/stop (may later also resume/suspend) functions that take "something". Nothing is stopping a developer from doing something like:

(ns billing
  (:require [db]))

(defn load-entitlements []
  (db/select ...))

(defstate entitlements :start (load-entitlements))

where it is not immediately obvious that entitlements depend on db.

We can take all the namespaces that have defstates, and filter all things that required/used. But that does not quite work if a single namespace has multiple defstates :)

needs more thinking..

[yogthos]

Yeah, I see how this can get tricky. Just tracing how defstates depend on one another based on the namespace graph would be helpful I think. That way you can get the big picture and then navigate through the namespaces and see the details.

[tolitius]

not sure I understand, can you give an example?

[yogthos]

Oh what I was thinking is that you can track the order namespaces with defstate require each other and then write that out:

(ns app.http-server)

(ns app.db)

(ns app.queue)

and http-server relies on db and queue being started then could output something like

{app.http-server [app.db app.queue]
 app.db []
 app.queue []}

[tolitius]

makes sense. how do we know it is:

[server
  [db queue]]

and not

[server
  [db]
  [queue]]

or

[server]
[db]
[queue]

?

[yogthos]

err updated mine to use a map :)

[tolitius]

what I am missing is how do we know that app.http-server depends on app.db and they are not just unrelated states that tend to start one after another?

[yogthos]

I would infer that from app.http-server requiring app.db, but it's possible that there could be intermediate namespaces there in the chain. So it might need a bit more cleverness than just following one ns to another.

[yogthos]

I think all that needs stating is the order, so we know db would be started before server can start for example.

[yogthos]

another related idea might be to allow start to take an optional map that specifies the order components should be started in. That way the user could explicitly define the order, e.g:

(mount.core/start
  {'app.http-server ['app.db 'app.queue]
   'app.db []
   'app.queue []}

[tolitius]

a constant battle between the levels of control and freedom :)

if it is just the order would not it be:

(mount.core/start
  ['app.http-server
   'app.db
   'app.queue]

instead?

Order is an interesting idea beyond visualization. One of the projects I use mount for, has a couple of queue listeners that I'd like to start last. The way I currently do it is by requiring them last in the ns with -main :)

[yogthos]

I think it might be useful to express it as a tree, if server depends on both db and queue, but db and queue don't depend on each other then they could be started simultaneously.

[yogthos]

but yeah possibly overthinking this :)

[tolitius]

yea, I feel that it couples "dependencies" which are crafted manually here on top of what really happens at runtime with "order"..

I think we can start with this:

dev=> (states-with-deps)

({:name app-config, :order 1, 
                    :ns #object[clojure.lang.Namespace 0x6e126efc "app.config"], 
                    :deps ()} 

 {:name conn, :order 2, 
              :ns #object[clojure.lang.Namespace 0xf1a66a6 "app.nyse"], 
              :deps ([app-config #'app.config/app-config])} 

 {:name should-not-start, :order 3, 
                          :ns #object[clojure.lang.Namespace 0x28116b9d "check.parts-test"], 
                          :deps ([conn #'app.nyse/conn])} 

 {:name nrepl, :order 4, 
               :ns #object[clojure.lang.Namespace 0x2c134117 "app"], 
               :deps ([app-config #'app.config/app-config])} 

 {:name test-conn, :order 5, 
                   :ns #object[clojure.lang.Namespace 0x1924a9f4 "check.start-with-test"], 
                   :deps ([conn #'app.nyse/conn] 
                          [app-config #'app.config/app-config] 
                          [nrepl #'app/nrepl])} 

 {:name test-nrepl, :order 6, 
                    :ns #object[clojure.lang.Namespace 0x1924a9f4 "check.start-with-test"], 
                    :deps ([conn #'app.nyse/conn] 
                           [app-config #'app.config/app-config] 
                           [nrepl #'app/nrepl])})

what do you think?

[yogthos]

yeah that looks looks great, it makes it possible to see the startup order which I think addresses the original issue :)

[tolitius]

kept it open, since I still want to be able to do some kind of "real time system visual".

(states-with-deps) moved to mount.tools.graph and now they have "states of states" and include namespace in state names:

dev=> (require '[mount.tools.graph :as graph])

dev=> (graph/states-with-deps)
({:name "#'app.conf/config", 
  :order 1, 
  :status #{:started}, 
  :deps #{}}
 {:name "#'app.db/conn",
  :order 2,
  :status #{:started},
  :deps #{"#'app.conf/config"}}
 {:name "#'app.www/nyse-app",
  :order 3,
  :status #{:started},
  :deps #{"#'app.conf/config"}}
 {:name "#'app.example/nrepl",
  :order 4,
  :status #{:started},
  :deps #{"#'app.www/nyse-app" "#'app.conf/config"}})

it is still quite rudimentary in terms of dependencies, since:

• these are namespace dependencies (not the actual dependencies of the state, but of a namespace that this state belongs to / defined at)
• won't work for ClojureScript, since we can't rely on namespaces be there (i.e. :advanced)

but besides deps, it is a little ascii visual that maybe helpful

[yogthos]

ah sounds like a great idea, looking forward to it :)

[danielcompton]

https://github.com/hilverd/lein-ns-dep-graph/ might be helpful for some inspiration.