documentation, what is missing ?

[leonoel]

Please share suggestions about ways to improve documentation.

[ribelo]

The documentation is good, but only for someone who catches the concept.

However, for someone who feels reasonably proficient in core.async, it's like colliding with a wall. Everything is different, and it's a little hard to translate things, to missionary.

I don't have a problem with reading tests, but in this case they are not always readable either, especially since you first have to understand how the underlying macros work.

IMHO something like this would suffice, touching on all the features available and how to use them. https://github.com/clojure/core.async/blob/master/examples/walkthrough.clj

[martinklepsch]

Some ideas:

• A walkthrough of some common workflows with ClojureScript, e.g. subscribe to DOM events, start/end flows etc.
• A list of the various macros m/? m/ap and so on with descriptions of what they do, CLJ/S compatibility, and a "how to read them", i.e. what words to use to describe them

IMHO something like this would suffice, touching on all the features available and how to use them. https://github.com/clojure/core.async/blob/master/examples/walkthrough.clj

There are some tools (like transcriptor) to turn REPL sessions into tests, maybe that'd be a good format for this.

[steffan-westcott]

Regarding the overall structure of documentation for any software project, I've come across this pattern which has been applied to several high profile projects: https://documentation.divio.com/ (The video is worth watching). I believe adopting this pattern would help both beginners and experienced users well.

[xificurC]

AFAIK the creator of that documentation structure moved it to https://diataxis.fr/

[xificurC]

A cookbook (called how-to guides above) is what I miss the most. The current 3 guides are nice, but more, shorter examples of common patterns would help me personally.

@leonoel any reason you're asking for suggestions? Did someone complain or do you find the documentation lacking?

[leonoel]

any reason you're asking for suggestions? Did someone complain or do you find the documentation lacking?

Both. A significant part of the library is not covered at all by current docs, several users have asked for more resources, and the learning curve feels steep. Also I'd like to understand better what newcomers struggle with.

[xificurC]

I still struggle understanding how will my code run, most importantly when will it block and what to do when I don't want a task to block. I think it's typically with tasks, flows are easier on this front.

Some of the primitives (e.g. in happy eyeballs race of dfv) are not part of the documentation, which makes happy eyeballs harder to understand.

How about we start with a couple of wiki pages? Specter has this nice list of navigators that I visit often and find valuable and short explanations with simple examples. A second page could be a cookbook of patterns.

[JohanCodinha]

How to create a Task and maybe an example on how to interop with a js callback api.

[bsless]

This partially relates to documentation and partially to naming, but I find that runes and acronyms make it harder to grok the code, examples and documentation. Until I'm fluent, I have to keep reminding myself, sp is sequential process, ap is asyncrhonous-process, not to mention all the variations on pulling and forking. If they had clearer names I think it would go a long way towards making the code more approachable to new arrivals. There are also some fundamental concepts such as forking which could be clarified. Also, for people who learn more by following things up from first principles than by example, it could be beneficial to create a continuity between the task and flow specification to the primitives in the library. This issue might be partially related, too https://github.com/leonoel/flow/issues/1

[mjmeintjes]

I think it would be useful to have more examples of how to integrate missionary with existing libraries and tools (in addition to the current documentation on integrating with RxJava):

Some examples:

• Converting core.async channel to flow
• Converting promise to task
• Consuming a BlockingQueue
• CompletableFuture to task
• File IO
• HTTP

It might be worth adding a wiki or something similar for managing and working on these types of snippets and examples.

[bsless]

I think it would be useful to have more examples of how to integrate missionary with existing libraries and tools (in addition to the current documentation on integrating with RxJava):

Some examples:

• Converting core.async channel to flow
• Converting promise to task
• Consuming a BlockingQueue
• CompletableFuture to task
• File IO
• HTTP

It might be worth adding a wiki or something similar for managing and working on these types of snippets and examples.

To build off this point a bit, I think it can be generalized to the "edges of the system". How can things be turned to tasks and flows? How do I start a flow which does some IO?

This example from a slack thread is a helpful learning aid, not just useful for my particular case.

(defn poll [^KafkaConsumer consumer]
  (m/via m/blk (.poll consumer Long/MAX_VALUE)))

(defn forever [task]
  (m/ap (m/? (m/?> (m/seed (repeat task))))))

(->> (forever (poll consumer))
  (m/eduction (comp cat (map -value)))
  (m/reduce (fn [_ x] (println 'consumed x)) nil)
  m/?)

[bsless]

Trying to see if I've been paying attention in class, CompletableFuture to task should look something like:

(import '(java.util.concurrent Executor CompletableFuture)
        '(missionary.impl Thunk))

(defn cf->task
  [^CompletableFuture cf]
  (fn [success failure]
    (.handleAsync
     cf
     (reify java.util.function.BiFunction
       (apply [_ r e]
         (if (instance? Exception e)
           (failure e)
           (success r))))
     Thunk/cpu)
    (fn [] (.cancel cf true))))

(let [cf (CompletableFuture.)
      t (cf->task cf)
      success! (fn [res] (println 'yay! res))
      fail! (fn [e] (println 'Error! (ex-message e)))]
  (t success! fail!)
  (.complete cf 2))

(let [cf (CompletableFuture.)
      t (cf->task cf)
      success! (fn [res] (println 'yay! res))
      fail! (fn [e] (println 'Error! (ex-message e)))]
  (t success! fail!)
  (.completeExceptionally cf (Exception. "failed!")))

[bsless]

Other terms which are unclear after rereading the documentation and specification, or which just aren't clear how one should work with

• discrete vs. continuous flow
• back pressure
• forking
• reactor
• introducing concurrency into a flow

[leonoel]

Thank you all, lots of relevant ideas here.

Regarding the overall structure of documentation for any software project, I've come across this pattern which has been applied to several high profile projects: https://documentation.divio.com/ (The video is worth watching). I believe adopting this pattern would help both beginners and experienced users well.

AFAIK the creator of that documentation structure moved it to https://diataxis.fr/

A cookbook (called how-to guides above) is what I miss the most. The current 3 guides are nice, but more, shorter examples of common patterns would help me personally.

How about we start with a couple of wiki pages? Specter has this nice list of navigators that I visit often and find valuable and short explanations with simple examples. A second page could be a cookbook of patterns.

diataxis is a good framework and I want to endorse it, here are my thoughts :

• tutorials : I think the current tutorials are decent but we need more.
• how-to guides : I like the idea of using the github wiki as a cookbook. I've moved the 3 existing guides and set public write access, feel free to contribute !
• reference : I think the docstrings are the right place for it, especially in combination with cljdoc to improve reading experience. The contents need refreshing and elaboration, I'm going to make a global pass at some point.
• discussions / explanations : we have none yet but I plan to write some.

IMHO something like this would suffice, touching on all the features available and how to use them. https://github.com/clojure/core.async/blob/master/examples/walkthrough.clj

Some of the primitives (e.g. in happy eyeballs race of dfv) are not part of the documentation, which makes happy eyeballs harder to understand.

I'm not sure if it's a good fit for a tutorial, it would probably be rather long and I see more value in a synthetic reference. I'd like to have some kind of cheatsheet, however it's still unclear to me what it should look like.

A walkthrough of some common workflows with ClojureScript, e.g. subscribe to DOM events, start/end flows etc.

How to create a Task and maybe an example on how to interop with a js callback api.

Converting core.async channel to flow

Converting promise to task

Consuming a BlockingQueue

CompletableFuture to task

File IO

HTTP

To build off this point a bit, I think it can be generalized to the "edges of the system".

How can things be turned to tasks and flows?

How do I start a flow which does some IO?

These are definitely good fits for the cookbook.

However, for someone who feels reasonably proficient in core.async, it's like colliding with a wall. Everything is different, and it's a little hard to translate things, to missionary.

I still struggle understanding how will my code run, most importantly when will it block and what to do when I don't want a task to block. I think it's typically with tasks, flows are easier on this front.

There are also some fundamental concepts such as forking which could be clarified.

forking

discrete vs. continuous flow

back pressure

My current plan for explanation pages, roughly :

• imperative vs functional : switching mindset
• execution model : which thread runs my code
• forking : how it works, how to reason about it
• discrete vs continuous : back pressure and lazy sampling

reactor

Yes. I have a WIP tutorial on the reactor.

[leonoel]

A list of the various macros m/? m/ap and so on with descriptions of what they do, CLJ/S compatibility, and a "how to read them", i.e. what words to use to describe them

This partially relates to documentation and partially to naming, but I find that runes and acronyms make it harder to grok the code, examples and documentation. Until I'm fluent, I have to keep reminding myself, sp is sequential process, ap is asyncrhonous-process, not to mention all the variations on pulling and forking. If they had clearer names I think it would go a long way towards making the code more approachable to new arrivals.

I opened a separate issue, #44

[dspiteself]

A great thing to add to the documentation is what is functional effect and streaming system? why use functional effects? It could be on separate page or inlined at the top of the main page. These questions overlap with the documentation of tasks maybe all that is needed is a bridge from the words functional effect and task and a bit more why.

My answer would be something like below. A side effect is when a function relies on, or modifies, something outside its parameters to do something. For example: (println "Hello world!") A functional effect is value that describes an effect, without actually executing it. In missionary we represent function effects using tasks. For example creating a sequential process for the effect hello world (def hello-world-task (sp (println "Hello world!"))) now will not print to the console until the task is ran (m/? hello-world-task) will run the task hello-world-task which will print "Hello world!" to the console. Using functional effects yields many benefits including improvement to testability and composability.

[kawas44]

Hi,

I have added a walkthrough "à la core.async" focused on Missionary Clojure implementation. The last example about flow is an implementation of the example from the core.async walkthrough using alts!!.

I have added a cheatsheet "à la specter", for the moment only about tasks and flows (I dont understand yet how to properly use dfv, mbx, sem, etc)

[xificurC]

@leonoel

I have a WIP tutorial on the reactor

could you share it even in its raw state?

[leonoel]

@kawas44 Thank you, these resources are highly valuable. I have some remarks about the walkthrough :

• (m/ap (for [i (range 4)] (m/?> i))) this is invalid code (because i is not a flow, and because m/?> is not allowed in closures)

• in the last example, the ap block is actually fully synchronous and single threaded. We could make it async using via, the execution would be closer to what happens with core.async.

  (let [begin (System/currentTimeMillis)
    ;; create a flow of values generated by asynchronous tasks
    inputs (repeat 1000 (m/via m/cpu "hi")) ;; a task has no identity, it can be reused
    values (m/ap
             (let [flow (m/seed inputs)     ;; create a flow of tasks to execute
                   task (m/?= flow)]        ;; from here, fork on every task in **parallel**
               (m/? task)))                 ;; get a forked task value when available
  
    ;; drain the flow of values and count them
    n (m/? ;; tasks are executed, and flow is consume here!!
       (m/reduce (fn [acc v]
                   (assert (= "hi" v))
                   (inc acc))
                 0 values))]
  (println "Read" n "msgs in" (- (System/currentTimeMillis) begin) "ms"))

[kawas44]

• (m/ap (for [i (range 4)] (m/?> i))) this is invalid code (because i is not a flow, and because m/?> is not allowed in closures)

Oups, didn't try to reduce it :grimacing: I can fix it with amb> which I understood while writing the cheatsheet :)

(m/ap (println (m/seed [1 2])))

EDIT: Changed the m/ap snippet above to use println instead of m/amb> to avoid introducing this macro early in the walkthrough.

• in the last example, the ap block is actually fully synchronous and single threaded. We could make it async using via, the execution would be closer to what happens with core.async.

Yes I knew it run on the same thread. I am still a little bit lost on the vocabulary. Maybe "concurrent" would have been a better word. Anyway, if you dont mind, I will copy/paste your version.

[dustingetz]

IMO any/all tutorials need to work in ClojureScript (so m/? outside process block should not be in the tutorial, instead describe the task interface and how to run it directly)

(def task (fn [success! failure!] (success! 1) (fn cancel! [])))
(def cancel! (task #(prn ::success %) #(prn ::failure %)))

also see https://github.com/leonoel/task

[leonoel]

@dustingetz makes sense to me, it also prevents a potential confusion between parking m/? and blocking m/?.

[xificurC]

hello task suffers the same issue btw. It even mentions the task works in cljs (which it does), but the blocking m/? call is misleading there too

[leonoel]

@xificurC

could you share it even in its raw state?

I'd like to make a step-by-step guide to solve the petrol pump problem popularized by the sodiumFRP team.

My current implementation relies on unreleased features, you'll need to compile the master branch locally.

[dustingetz]

(tests
  "Missionary signals"
  (def !x (atom 0))                                         ; atoms model variable inputs
  (def >x (m/watch !x))                                     ; "recipe" for a signal derived from atom

  ; a signal is a "continuous flow" in Missionary jargon
  ; signals (flows) are recipes, like Haskell IO actions.
  ; Like IO actions, they are pure function values (thunks)
  ; and do not perform side effects until you run them.
  (fn? >x) := true                                          ; thunk (implementation detail)
  ; The atom has not been subscribed to yet, because >x is a pure value

  ; Flow thunks concretely have the structure (fn [notify! terminate!] !iterator),
  ; see https://github.com/leonoel/flow#specification
  (def !it (>x (fn [] (! ::notify))
               (fn [] (! ::terminate))))
  % := ::notify                                             ; lazy flow is ready to be sampled
  @!it := 0                                                 ; sample
  (swap! !x inc)                                            ; trigger a change
  % := ::notify                                             ; flow is ready again
  @!it := 1                                                 ; sample
  )

[PEZ]

@dustingetz ! in those tests does not resolve for me. Do I need to refer something? m/! expects zero arguments...

[PEZ]

I specifically started to browse the various resources to try to figure out if I can compose flows. I have a continuous flow that I want to build another continuous flow from.

I then quite quickly realized I needed to start from first principles instead. Though I think I need a combination of the concepts laid out and walkthroughs that I can run in the REPL. I like how the Tasks & Flow walkthrough starts. 😄

[dustingetz]

! there is hyperfiddle.rcf/tap, which used to be hyperfiddle.rcf/!

[awwx]

The example at the top of the README uses signal:

(let [<x (m/signal (m/watch !input)) 
      ...

What does signal do? https://cljdoc.org/d/missionary/missionary/b.32/api/missionary.core#signal says signal returns a publisher.

What is a publisher?

https://cljdoc.org/d/missionary/missionary/b.32/api/missionary.core#publisher says "Returns a org.reactivestreams.Publisher running given discrete flow on each subscription." What does that mean?

reduce takes a flow, and in the README example is passed the publisher returned by signal. Is a publisher a kind of flow? What makes it different from other kinds of flows, such as the flow returned by watch?

https://github.com/leonoel/missionary/wiki/Continuous-flows#sharing says to use signal when sharing: "Make sure sharing happens in a reactor context and use signal!" Why? (What happens if you don't use signal when sharing?)

https://github.com/leonoel/missionary#vs-reactive says that "the flow abstraction, a foundational protocol allowing a producer to signal availability of a value without eagerly computing it". Is the signal referred to here the same kind of signal as what signal does?

It looks like https://github.com/leonoel/task has useful details on the task abstraction, but I didn't see it linked from the missionary documentation (maybe I missed it).

Likewise it looks like https://github.com/leonoel/flow has useful details for understanding the flow abstraction.

Returning to https://cljdoc.org/d/missionary/missionary/b.32/api/missionary.core#signal, the documentation there talks about subscribers and subscriptions, but those aren't mentioned in the https://github.com/leonoel/flow spec.

In https://github.com/leonoel/flow:

A flow is a function provided by the producer taking two arguments, a notifier and a terminator. It is called by the consumer to spawn a new instance of the process.

I'm not clear by reading this far whether "it" is referring to the flow or the producer. (Reading further I deduce "it" here refers the flow function).

I'm not clear from the spec whether the flow function can be called more than once, and what happens if it is. Is each call to the flow function a subscription? What happens if one subscription is made, some values are produced, and then a second subscription is made? Are the previously produced values sent to the second subscription, or only new values, or is that unspecified?

[leonoel]

@awwx Thank you, this is a valuable feedback.

It is allowed to spawn the same effect (task or flow) more than once, it will result in performing the actions described by this effect more than once. A publisher is a memoized view of an effect, it is also an effect and the action it performs is called a subscription. Running a publisher twice registers two subscriptions, but the underlying effect is spawned only once due to memoization.

I acknowledge the wording confusion. Please note :

• the wiki is outdated. reactor, signal! and stream! are deprecated and superseded by signal and stream (without bang suffix)
• publisher and subscribe are about reactive streams interop and orthogonal to missionary publishers. The name collision is unfortunate, but the interface Publisher in reactive streams is not necessarily about publishing as it's merely an effect descriptor (like flow)
• the documentation is currently being rewritten, there will be a detailed explanation about the effect protocols and the new publisher design, I will make sure the terminology is consistent and up to date.

[awwx]

ok, as an introduction in the documentation, perhaps something like the following (please feel free to use any of this in the documentation, if it's correct and useful):

An instance of a flow transfers a series of values from a single producer to a single consumer. (Flows can be multiplexed if needed by using a Missionary publisher).

If computers were infinitely fast, a consumer would always be able to keep up with its publisher. Since sometimes a consumer might not able to keep up, we need a strategy for what to do when it doesn't. Two common strategies are:

• We have a flow of discrete values, such as events, where it's important to process every value (and we hope that the consumer falling behind is only temporary). Then either the values need to only be generated when the consumer is ready to receive them (back pressure), or, if the values have already been generated, they need to be retained so that they can be delivered when the consumer is ready.

• We have a flow for a continuous signal, where when the consumer is ready to receive a new value, the consumer only cares about the most recent value of the signal. If the consumer has fallen behind, old values can be discarded.

For example, m/seed returns a discrete flow from a Clojure collection, and each item in the collection will be delivered to the consumer when the consumer is ready to receive it, regardless of how long the consumer takes to receive the values.

m/watch returns a continuous flow reflecting the current value of a Clojure reference type. As long as the consumer can keep up, each update of the reference type will be delivered, but, if the consumer falls behind, when the consumer is ready to receive again it will get the latest value of the reference type without having seen intermediate values.

Both scenarios are accommodated in the flow specification by using a handshake protocol. The consumer notifies the producer when its ready to receive a new value. Neither the producer nor the consumer gets ahead of the other: the producer waits until it's been notified by the consumer before sending the consumer a new value, and the consumer waits until it's received a value before notifying the producer again.

[awb99]

I suggest that depreciated functions/macros are NOT included in the cljdocs.

This would make sense because once something gets depreciated, then new users (who would read the clj-docs) should not use it, so it should not be in the clj-docs.

This is important because missionary has a lot of operators and it is not easy to get ones head around to find the right one. If the cljdoc list gets reduced a little, then this helps.

Thanks!