Parallel t.n.r/refresh

[vemv]

Problem

• The Clojure compiler is slow, unavoidably, as of today
  • proof: (dotimes [_ 100] (time (require 'hello-world :reload)))
  • even if it takes a few millis, that's slow at scale
• refresh does its job sequentially.

Consequently, large projects will have a slow refresh time, as its cost scales linerarly.

Solution

It might be possible to implement a parallel refresh, on top of the t.n library.

Considerations

• require is not thread safe, so a naive impl would cause issues.
• Accordingly, the work should be partitioned in 'islands' (see example below), which are provably independent subsets of the project's namespaces
• Still, just in case the library should have an internal lock around require usage.
• For 'island' analysis, ns parsing isn't enough: there can be manual require, resolve etc calls below the ns (or simply ad-hoc calls to other namespaces, with no requiring at all).

Example

For a project with 4 namespaces:

(ns foo)

(ns depends-on-foo
  (:require
   [foo]))

(ns bar)

(ns depends-on-bar
  (:require
   [bar]))

There are two 'islands':

• foo + depends-on-foo
• bar + depends-on-bar

So, each island can be loaded from a separate thread.

Note, as per t.n's tracker, maybe only e.g. bar should be reloaded. So one doesn't blindly reload islands: one composes them with the tracker.