--server mode

[thheller]

Starting with [thheller/shadow-cljs "1.0.20170613"] and shadow-cljs@0.9.6 I added a proper --server mode.

There are several things this is meant to solve

• running builds in parallel
• less annoying REPL
• controlling builds remotely
• possibly a nice Web UI (in the future)

By default it starts a HTTP Server at {:host "localhost" :port 8200} (configured via the :http key in the config). As of today this does nothing useful other than being the endpoint for the devtools websockets.

It also starts a Clojure Socket REPL at {:host "localhost" :port 8201} (configured via the :repl key in the config). This is not nREPL but a plain Clojure 1.8 Socket REPL which means anything that is able to connect to a TCP port can be used as a client. rlwrap nc localhost 8201 is the most basic example of such a client.

Running shadow-cljs --server looks something like this

shadow-cljs - 1.0.20170613 using /Users/zilence/code/shadow-cljs-examples/cljs-react-app/shadow-cljs.edn
shadow-cljs - updating dependencies
Retrieving thheller/shadow-cljs/1.0.20170613/shadow-cljs-1.0.20170613.pom from http://clojars.org/repo/
Retrieving thheller/shadow-cljs/1.0.20170613/shadow-cljs-1.0.20170613.jar from http://clojars.org/repo/
shadow-cljs - dependencies updated
shadow-cljs - optimizing startup
shadow-cljs - starting ...
shadow-cljs - server running at http://localhost:8200
shadow-cljs - socket repl running at tcp://localhost:8201

If you now connect to the Socket REPL via rlwrap nc localhost 8201 you should be greeted with some intro text and a REPL prompt

shadow-cljs - CLJ REPL

(shadow/start-worker :foo) - Starts a dev process for build :foo
(shadow/repl :foo) - Switches the current REPL to that of :foo
(shadow/stop-worker :foo) - Stop the dev process

(shadow/once :foo) - Compiles a dev build once
(shadow/release :foo) - Create a release build for :foo

More coming soon ...
[1:0] shadow.user=>

It is a normal Clojure REPL that provides the above utility functions but all Clojure works. To upgrade this REPL into a CLJS REPL you first need to start a worker by calling (shadow/start-worker :npm)

[1:0] shadow.user=> (shadow/start-worker :npm)
:started

All output generated by this worker is displayed in the window of the --server process as I think it is way too annoying to mix the REPL with build logs.

You can connect to the CLJS REPL whenever you need to by calling shadow/repl

[1:0] shadow.user=> (shadow/repl :npm)
[1:1] cljs.user=>

Your REPL is now upgraded to a CLJS REPL. Type :repl/quit to drop back down to the normal CLJS REPL. :repl/quit also exits the CLJ REPL and disconnects the socket. The --server will remain running.

There is one extra new config entry :server {:autostart #{:foo :bar}} which lets you autostart a few dev workers when starting the --server process as a convenience.

[thheller]

@arichiardi the Socket REPL should work with inf-clojure but I'm not sure how to hook up the auto-complete stuff yet. Briefly looking over inf-clojure source looks like there are special cases for lumo and planck already. I hope to just use the default ...

I'll probably also add nREPL support so cider can work but it such a huge project that I don't even know where to start. nREPL middleware scares me. Cursive currently also can only connect to a remote nREPL.

[pedrorgirardi]

@thheller is there any kind of integration with any text editor - VSCode, Atom, Emacs or even Cursive? What would be a nice setup for ClojureScript development with shadow-cljs?

Is there a place for discussing things like this?

[thheller]

@pedrorgirardi I hope to get that some day but it is a bit too early. FWIW I'm using Cursive and it sort of works the only real pain point is that it can't connect to a Socket REPL remotely.

shadow-cljs can also use lein directly, so :source-paths and :dependencies from lein are used. https://github.com/thheller/shadow-cljs/blob/master/shadow-cljs.edn is the setup I use during development. shadow-cljs will activate the :cljs profile defined in project.clj and go through that. Obviously it would be nicer if tools could just understand shadow-cljs.edn but that will probably take some time and broader adoption of shadow-cljs.

I'm probably doing the nREPL next which should enable things like cider and whatever atom or vscode have. emacs inf-clojure-connect seems to work but I think it has trouble with the non-standard [4:0] shadow.user=> prompt. I saw that you can change the regexp for that but my emacs-fu is very limited and I don't know how.

[pedrorgirardi]

Thanks @thheller! Unfortunately Clojure-Kit can't connect to a remote socket REPL either, but a remote nREPL would work. I wonder if it's going to take much time for editors to start supporting socket REPL.

I don't have development experience with tooling and the kind of thing that you are doing with shadow-cljs, but I'm happy to help with issues and trying things out and sharing with people about the great work that you are doing with shadow-cljs. I believe that a project like this is very valuable for the community and for newcomers.

[thheller]

Thanks @pedrorgirardi. All feedback helps. Documentation is the most important thing at this time and I never know what (or how) to write it since I don't know what questions people may have. I know how the thing works as I have been using it for 3+ years.

Just ask anything really, ideally with examples if something is unclear. I need to know what people are doing so I can make proper guides. I want to collect some form of FAQ and go from there.

[tiye]

I got questions. Why I still need a server when I can use command line tools? As you mentioned WebSocket, what protocol does it use?

[thheller]

I need to do a full write-up about --server but it is basically a super --dev mode.

The intent is that you start that ONCE and leave it running somewhere. It can then be used to actually do all compilations which means every command does not need to start a new JVM but instead can use the --server process instead which basically cuts script start up time to almost zero. The server can also be embedded if you have a lein repl running anyways. I always have that running for Clojure stuff anyways. When I have 3 builds running for work embedding the shadow-cljs server cuts the number of running JVMs from 5 to 1. I have only used the embedded version myself, I have never used the CLI tool for work. CLI is nice to get started by --server (standalone or embedded) will always be the endgame, but fully optional if you don't want it.

--dev processes currently are basically their own server, you just can't talk to it from the outside. You need a server somewhere to do the file watching and REPL. The :devtools will attempt to connect to the websocket provided by that server to do the live-reload/REPL.

This will have 2 running JVMs that don't share anything.

shadow-cljs --build foo --dev
shadow-cljs --build bar --dev

Sometime soon if there is a --server running all CLI commands will connect to it instead of launching a new JVM. This removes the JVM startup time, ie. --once should take less than a second when cached. Currently it takes at least 4sec.

--server will also provide an endpoint that tools can talk to. There will be a formal API that an editor/IDE can use. The endgame also includes Language Server Support which is a "standard" API for completions and the like.

I did a proof of concept a while back to integrate warnings generated by the compiler directly with the editor. (This is an actual screenshot from VSCode)

I think it is far cooler if warnings show up directly in your editor when you save the file instead of having to look at a terminal/browser somewhere.

https://code.visualstudio.com/blogs/2016/06/27/common-language-protocol https://github.com/Microsoft/language-server-protocol/wiki/Protocol-Implementations

Not a whole lot of editors support it currently but I hope that will change at some point. There is neovim and emacs support already. Things like cider-nrepl basically implement this functionality over nREPL and you need a different implementation for every other editor. LSP could be a solution that works for multiple editors if you implement it once and it will require --server. It would cover completions, doc lookup, source navigation and a bunch of other stuff. LSP is not all great but better than nREPL for tooling (IMHO, YMMV).

[thheller]

I have a question: Should the --server process be a REPL itself?

--server currently starts a standalone process that will basically just log build output and show warnings/errors. You can only do something to that process via remote but not from the terminal it is running in.

I could start a REPL for the --server process itself so you can interact with it directly and only optionally connect to it remotely. I opted NOT to do that because I hate REPLs that basically always have a broken prompt. If you type into the REPL and log output shows up it is going to be all garbled and I hate that. But when starting the server it doesn't do much and you'd have a clean prompt.

Currently you have to start the server and then connect to it from somewhere else. Tools will always do that anyways but from a UX experience it might make sense to have a REPL directly available. figwheel also does that.

[tiye]

How about this case? I use shadow-cljs to build my code, it calls shadow-cljs twice:

  "scripts": {
    "build": "yarn del && yarn cljs-release && yarn webpack && yarn cljs-once && yarn html && yarn serve"
  }

And it's started by:

yarn build

What's the designed solution to solve this? Do I have to run --server somewhere else and create my own triggers for that?

[thheller]

--server will be optional and the CLI will just use it if its running. shadow-cljs --build app --release will have the exact same result whether --server is running or not. The only difference will be that it starts faster and uses less memory. This is not yet implemented, coming soon.

Not sure I understand the intent here. Why do a release and then immediately once?

Are you working around the problem that release files overwrite dev files and vice versa? If thats the case you should probably use a different :output-dir for :release so it doesn't interfere with :dev? All this is currently a bit annoying with webpack but I think you can override where it will look for files via resolve.alias or resolve.modules so your actual require doesn't have to change.

It is all a bit dirty with :npm-module currently since I haven't yet figured out how people usually do this kind of stuff in the JS world.

[tiye]

Agreed, I need another :output-dir to separate compiled files from two targets in future versions.

I need to compile twice, once for the app itself, the other one is to do building-time-server-rending of HTML files. In JavaScript world there are always dirty solutions for finding shortcuts, I think.

[thheller]

You are also still mixing 2 different builds into one.

{:source-paths ["src"]
 :dependencies [[mvc-works/hsl "0.1.2"]
                [respo/ui "0.1.9"]
                [respo "0.4.5"]]
 :builds {:app {:output-dir "target/"
                :target :npm-module
                :release {:entries [app.main]}}
          :render {:target :node-script
                   :output-to "scripts/render.js"
                   :main app.render/main!}}} ;; remove the direct (main!) call, the script will call it 

node scripts/render.js afterwards and you only need to run shadow-cljs --build render --once whenever you change the render script. (or run it with --dev, it won't interfere with :app at all.

[tiye]

Will try :P

[thheller]

Just pushed the experimental remote mode I mentioned. If you launch shadow-cljs --server and leave that running somewhere any other shadow-cljs calls will connect to that server and invoke the command there.

All commands should yield the same results whether the server is running or not. It just will be a lot faster when the server is available.

Without --server

time shadow-cljs --build app --once
shadow-cljs - 1.0.20170615 using /Users/zilence/code/shadow-cljs-examples/code-split/shadow-cljs.edn
shadow-cljs - starting ...
...
real    0m4.318s
user    0m12.388s
sys 0m0.534s

With --server running

time shadow-cljs --build app --once
shadow-cljs - 1.0.20170615 using /Users/zilence/code/shadow-cljs-examples/code-split/shadow-cljs.edn
shadow-cljs - connected to server
...
real    0m0.731s
user    0m0.093s
sys 0m0.025s

There is also shadow-cljs --repl if you just want to connect to the server and play with the REPL. rlwrap nc localhost 8201 also works, as should everything that supports Clojure 1.8 Socket REPL.

[arichiardi]

Is --server creating a socket repl or only --repl does it?

[thheller]

--server is creating the socket, --repl connects to it. Would --connect make this clearer?

I still need to update --help.

[arichiardi]

There is no name convention now for creating a repl, in boot it is called -s for the nRepl server and -c (client). But this is within the repl task so you already have the context there. Do you have also a way to start a server and connect a client to it?

You could do something like:

• --repl launches both server and client (akin to boot repl or lein repl)
• --repl socket or --repl client - customize the behavior and launch what specified

The above is nice because if you want to get fancy at some point you can go for --repl node directly or --repl nrepl-server.

[thheller]

Things are a bit more complicated then I'd like but I'm not sure how to make it easier.

--server always starts the http server, socket repl and nrepl at some point. This will only be configurable via shadow-cljs.edn. IMHO it does not make sense to also create command line options for this. It is not just about the socket-repl here, it will also provide the endpoint for tools to talk to and the web UI.

This process could easily run a REPL itself but the reason I don't do that is because of the spurious printing it may do. The --server process currently prints build progress messages, warnings and basically everything else. It would be useless trying to parse this with something like inf-clojure.

The confusing part here is that every new REPL connection starts out as a CLJ REPL. Exposing a dedicated TCP Port for each CLJS REPL would be possible but a nightmare to keep track off and configure.

In #52 I mentioned that I intent to change the CLI options so you could shadow-cljs --repl build-id to directly connect to a CLJS REPL for the given build, skipping over the CLJ REPL.

We do however need the CLJ REPL to control shadow-cljs itself, eg. start/stop builds.

It will all get easier with the web UI I want to build but thats going to take a while and the CLI must exist anyways for shell scripts and stuff.