Current streams impl only works with Node (maybe use Highland instead?)

[jsa-aerial]

Hi,

I can now pull cljs-msgpack-lite into project with new simple dependency and simply

(ns ...
  (:require [cljs-msgpack-lite.core :as mpk]
                [cljs-msgpack-lite.clj-codec :refer [create-clj-codec]]) 

At this point, the resource appear as available - even with automatic completion, arg doc, and doc strings (working nicely from cider even). Looking great!

But then invoking them causes errors. (mpk/encode 1) => #object[TypeError TypeError: Cannot read property 'call' of undefined]

(def clj-codec (create-clj-codec)) =>

object[RangeError RangeError: Maximum call stack size exceeded]

cljs.core.PersistentArrayMap.cljs$core$ILookup$_lookup$arity$2 (jar:file:/home/jsa/.m2/repository/org/clojure/clojurescript/1.10.238/clojurescript-1.10.238.jar!/cljs/core.cljs:6733:1) Function.cljs.core.get.cljs$core$IFn$_invoke$arity$2 (jar:file:/home/jsa/.m2/repository/org/clojure/clojurescript/1.10.238/clojurescript-1.10.238.jar!/cljs/core.cljs:1923:30) cljs.core/get (jar:file:/home/jsa/.m2/repository/org/clojure/clojurescript/1.10.238/clojurescript-1.10.238.jar!/cljs/core.cljs:1917:1) cljs$spec$alpha$accept_nilQMARK (jar:file:/home/jsa/.m2/repository/org/clojure/clojurescript/1.10.238/clojurescript-1.10.238.jar!/cljs/spec/alpha.cljs:1010:3) cljs.core/every? (jar:file:/home/jsa/.m2/repository/org/clojure/clojurescript/1.10.238/clojurescript-1.10.238.jar!/cljs/core.cljs:14434:3) cljs.spec.alpha/accept-nil? (jar:file:/home/jsa/.m2/repository/org/clojure/clojurescript/1.10.238/clojurescript-1.10.238.jar!/cljs/spec/alpha.cljs:1019:15) cljs.spec.alpha/re-conform (jar:file:/home/jsa/.m2/repository/org/clojure/clojurescript/1.10.238/clojurescript-1.10.238.jar!/cljs/spec/alpha.cljs:1205:10) cljs.spec.alpha.regex_spec_impl.cljs.spec.alpha.t_cljs$spec$alpha41223.cljs$spec$alpha$Spec$conformSTAR$arity$2 (jar:file:/home/jsa/.m2/repository/org/clojure/clojurescript/1.10.238/clojurescript-1.10.238.jar!/cljs/spec/alpha.cljs:1252:10) cljs$spec$alpha$conformSTAR (jar:file:/home/jsa/.m2/repository/org/clojure/clojurescript/1.10.238/clojurescript-1.10.238.jar!/cljs/spec/alpha.cljs:40:14)

Am I missing something to do first, or did something get broken?

[jsa-aerial]

Ugh, it (cljs and/or js) doesn't like the require in this:

(def node-stream (js/require "stream"))

in cljs-msgpack-lite.core at line 8

Error in js console:

core.cljs?rel=1527971488643:8 Uncaught ReferenceError: require is not defined at core.cljs?rel=1527971488643:8 (anonymous) @ core.cljs?rel=1527971488643:8

I'm guessing that form is no longer needed with the cljsjs version

[christo-auer]

Thanks for the info! Strange, doo tests run perfectly on my machine, within a nodejs environment though. What environment do you use?

I've just uploaded a new version in which I require nodejs and "require" explicitely. Alternatively, I could extract the stream stuff from the core and keep it nodejs agnostic.

[jsa-aerial]

OK, I think I know pretty much what is going on here. Noodling around I see that the 'stream' in question is a nodejs lib https://nodejs.org/api/stream.html

You are running in a nodejs environment (targeting node), and likely have this lib installed via npm (like how you previously were using msgpack-lite.js) This is why you can run this without error.

I am running in/targeting the browser - which is why the new version pulling in nodejs.cljs doesn't work for me either - per the documentation for it, you can only use nodejs.cljs if you are targeting node in order to get node's external globals (like require). Since, I'm in the browser, this won't work for me.

Looking a bit more, I see that the stream lib has not yet been put into CLJSJS packaging, which would have solved the entire problem as you could just require it like you now require msgpack-lite from CLJS's NS form.

Having the stream stuff is truly useful - and will at some point be needed by me as well. So, pulling it out seems unfortunate at best. But using the nodejs.stream lib may only work in nodejs. OTOH, there is this https://highlandjs.org/ which purports to be both node and browser ready.

Hang on, maybe all is not lost. Short of figuring out how to make the CLJSJS package for stream - which would be the best solution, I just don't know how to do that - there are new facilities in later CLJS versions (apparently post 1.9.854) that are supposed to make this work fairly simply https://clojurescript.org/news/2017-07-12-clojurescript-is-not-an-island-integrating-node-modules Since you are using 1.9.908 this should be available. Hmmm reading that a bit more closely it appears that it can magically make these things work in the browser as well: "It’s worth noting that the module processing feature in ClojureScript is mostly intended to be used in projects that target the browser". Maybe 'requiring' stream using this facility would work after all??

[jsa-aerial]

Asking about use of nodejs.streams in browser on slack gave this answer: "that’s a native Node.js API, it does not work in the browser. you could possibly find a JS library that implements something similar that works in the browser"

So, https://highlandjs.org/ is probably the best way forward if you want to make this work in both Node and Browser settings.

Given that, maybe it would be better to start by doing what you originally suggested: "I could extract the stream stuff from the core and keep it nodejs agnostic". Then maybe we could try to convert the stream stuff to use highland (which seems to inter operate with node.stream streams, based on the examples they show).