clj-commons / manifold

A compatibility layer for event-driven abstractions
1.02k stars 106 forks source link

manifold.executor/with-executor doesn't override all execution #186

Closed positron closed 3 years ago

positron commented 4 years ago

From what I can tell from the documentation, there is no way to provide a custom executor if an md/chain starts on the wait-pool. To reproduce:

  (def my-executor
    (reify java.util.concurrent.Executor
            (execute [_ runnable]
              (log/info "inside custom executor")
              (.execute (me/execute-pool) runnable))))

When I use md/future it uses the provided executor.

  (me/with-executor my-executor
    (md/chain (future (Thread/sleep 100) "asdf")
              (fn [x] (log/info x))))
=>
INFO[user:1] - inside custom executor
INFO[user:1] - asdf

When I use clojure.core/future it doesn't.

  (me/with-executor my-executor
    (md/chain (future (Thread/sleep 100) "asdf")
              (fn [x] (log/info x))))
=>
INFO[user:1] - asdf

Background on my use case in case I'm just approaching this from the wrong direction: For logging purposes, everywhere I use manifold I want to use an executor that will copy some thread local data to the new thread. Because we use manifold extensively throughout our code base I want to just use the with-executor wrapper in some ring middleware as opposed to modifying call sites all over the place to use a custom md/onto, or some bespoke md/chain wrapper, or any similar solution.


manifold 0.1.8

mping-exo commented 4 years ago

I think that although manifold is compatible with clojure futures, they use their own internal threadpool.

KingMob commented 3 years ago

@positron As @mping-exo said, Clojure's thread fns use their own separate executors, not the Manifold wait-pool, and your ability to control them is sadly minimal. (This is why Manifold is built on top of Dirigiste, which offers more tuning knobs.)

Your best bet is to avoid Clojure's future if you want more control over the thread pools.

That being said, I'm a little confused by "... I want to use an executor that will copy some thread local data to the new thread." Clojure's future and core.async's thread do binding conveyance, which sounds like what you want. If you declare a var ^:dynamic and override it in each thread, the local binding will be "copied" to child threads started with future or thread. (Manifold's future and future-with do the same.)

Closing for now, as this isn't something fixable in Manifold.