Send messages over nREPL for a variety of events

[vemv]

As a preface, over the last two weeks I've worked quite a lot on a less buggy CIDER<->shadow-cljs integration. We have a new release around the corner which will bundle those.

After that work, I've come to observe that my recently created issues (https://github.com/thheller/shadow-cljs/issues/1138, https://github.com/thheller/shadow-cljs/issues/1136, https://github.com/thheller/shadow-cljs/issues/1135) all have have one thing in common: I want to know when stuff happened, and react to it.

From https://github.com/thheller/shadow-cljs/issues/1135 and https://github.com/thheller/shadow-cljs/issues/1136 feedback, I sense that it could be a hard sell for shadow-cljs to react to conditions (e.g. reload an .html page), but CIDER could happily do those.

For that, we can leverage the fact that a given nREPL message can have multiple, async responses over an indefinite period of time.

For example, for this message:

(-->
  id         "10"
  op         "eval"
  session    "62cffa12-b4ad-4517-9223-1478ca773f69"
  time-stamp "2023-08-07 09:22:38.510263000"
  code       "(do (require '[shadow.cljs.devtools.api :as shadow]) (shadow/watch :app) (shadow/nrepl-select :app))"
  ns         "shadow.user"
)

shadow-cljs could respond each time something interesting happened.

Those include:

• connected
• disconnected
• compiled successfully
  • with 0/N warnings
• failed to compile
  • with the errors foo bar baz
• stale output detected

Example message:

(<--
  id                 "10"
  session            "62cffa12-b4ad-4517-9223-1478ca773f69"
  time-stamp         "2023-08-07 09:22:39.626539000"
  status             ("shadow-cljs/compiled")
  shadow-cljs/compilation-info {:files 324 :warnings 0 :time-ms 10}
)

Using this info, CIDER and others can build better UIs. For instance, currently the compilation info is printed to stdout, but:

• It's displayed at the REPL, which is not a first-class UI for this type of output
  • nowadays CIDER, by default, doesn't even scroll the repl on new stdout (for performance reasons)
  • if we had an event, we could instead choose not to render that piece of stdout/stderr, and inform the user via a colorful UI instead.
• We have to parse sdout/stderr for knowing when a compile succeeded
  • (knowing when a compile succeeded is useful for rebuilding our ns cache)
  • parsing each line for stdout/stderr for a possible shadow-cljs message feels quite fragile and inefficient.

I hope this sounds interesting - please let me know what you think!

Thanks - V

[thheller]

Most is probably already available. Just a tad different than you are proposing here. I opened https://github.com/thheller/shadow-cljs/issues/990 for this but nobody showed any interest, so I closed it after a year. As described in that issue shadow.remote powers all remote interactions that shadow-cljs has. The entire UI runs over it, as does the nREPL implementation.

For example over nrepl send the following:

{:op "shadow-remote-init" :data-type "edn"}

That just sets up the connection, required to send this once before anything else.

Then you can send (no clue how you'd do an actual pr-str in cider)

{:op "shadow-remote-msg"
 :data (pr-str {:op :request-clients
                :notify true
                :query [:eq :type :runtime]})}

;; so this, not a literal (pr-str ...)

{:op "shadow-remote-msg"
 :data "{:op :request-clients :notify true :query [:eq :type :runtime]}"}

• :request-clients will cause the relay to send you all clients it knows about.
• :notify controls whether you want to be notified about changes (disconnect/connects)
• :query is a very simplistic edn based query language

This example you only get clients that identify as a :runtime (aka CLJ(S) runtimes), otherwise you also get yourself and the UI connections and so on. Which is fine but probably not something you are interested in. You can also limit it to specific builds, but the client info will also contain the build-id so sending this once is probably better than sending it per build.

On the receiving end you'll get {:op "shadow-remote-msg" :data "edn-encoded-string"} messages, that should be self explanatory. There will be a :op :welcome message, and the result for :request-clients is just :op :clients and :op :notify for updates.

I don't know what the state of EDN encoding/decoding is in cider, another available option is :data-type "transit". shadow.remote only understands EDN type data, so JSON/bencode or others are not supported and I'm not really interested in adding support for them, since shadow.remote needs actual EDN data (eg. keywords).

I can list some more common :op (e.g. eval, getting build infos, etc) if this is something that would work for your usecase. Since shadow-cljs uses this extensively itself pretty much everything is available, although often optimized for what the UI needs.

[thheller]

You could also just skip nrepl and directly connect to the websocket if cider/emacs supports that, nREPL really doesn't bring much to the table in this regard. The messages you'll exchange are the same though, so doesn't matter too much.

[vemv]

https://github.com/thheller/shadow-cljs/issues/990 sounds fully workable to me.

I'm not sure about the need for pr-str. As far as I know, you can attach arbitrary data over an nREPL message, at least for Clojure's basic data types. 'Bencoding' would take care of the rest.

But pr-str is certainly OK. You can send edn-encoded strings, and we would decode it CIDER side with https://github.com/clojure-emacs/parseedn which is a trusted solution.

You could also just skip nrepl and directly connect to the websocket if cider/emacs supports that

Emacs being such a quirky piece of software, I'd prefer it we sticked to nREPL so that we can reuse all our tech and knowledge.

Thanks!

-V

[thheller]

shadow-remote-msg is just an envelope and it expects an encoded field, as long as you send this its fine.

;; valid 
{:op "shadow-remote-msg"
 :data "{:op :request-clients :notify true :query [:eq :type :runtime]}"}

;; NOT valid 
{:op "shadow-remote-msg"
 :data {:op :request-clients :notify true :query [:eq :type :runtime]}}