Dynamic Binding

[myu07]

I get a binding exception when I'm invoking clojure.data.json/readstr in yarn-client mode: java.lang.IllegalStateException: Can't dynamically bind non-dynamic var: clojure.data.json/value-fn

I started the repl using: spark-submit --master yarn-client --class powderkeg.repl my.jar

In repl: (require '[clojure.data.json :as json]) (into [] (-> (vector "{\"a\":1,\"b\":2}" "{\"a\":4,\"b\":5}") (keg/rdd (map #(json/read-str % :key-fn keyword)))))

Looking at the clojure.data.json code, value-fn is a dynamic namespace variable. When clojure.data.json/read tries to rebind value-fn (a variable that is not locally defined within the function) it throws the binding exception Basically, it is aware of the variable's existence but does not see it as dynamically defined.

I was able to recreate the problem without using clojure.data.json by externally defining a dynamic variable and using it inside the anonymous function that gets submitted into the nodes.

In repl: (def ^{:dynamic true } divisor-ext 2)

(defn computeQuot1 [dividend divisor] (binding [divisor-ext divisor] (quot dividend divisor-ext )))

(into [] (-> [10, 15] (keg/rdd (map #(computeQuot1 % 5)))))

java.lang.IllegalStateException: Can't dynamically bind non-dynamic var: user/divisor-ext

However, if I defined the dynamic variable within the function that gets submitted into the nodes, it works fine:

In repl: (defn computeQuot2 [dividend divisor] (def ^{:dynamic true } divisor-int 5) (binding [divisor-int divisor] (quot dividend divisor-int)))

(into [] (-> [10, 15] (keg/rdd (map #(computeQuot2 % 10)))))

Is there any way we could use clojure.data.json (or any library that uses externally defined dynamic variable) in the cluster? FYI: this works in flambo (repl) in yarn-client mode.

[cgrand]

I published a 0.4.1 version including the fix, thanks!

[billnunes]

Thanks!

[cgrand]

Let us know how keg works (or not) for you!

[cgrand]

@billnunes I made a typo in 0.4.1, try 0.4.2 instead.

[myu07]

I tested the 0.4.2 version and it didn't completely fix the problem. If the dynamic variable was initialized during declaration, then everything works fine. However, if the dynamic variable is unbound, it fails with the same exception.

(def ^{:dynamic true } divisor-un)

(defn computeQuotUn [dividend divisor] (binding [divisor-un divisor] (quot dividend divisor-un )))

(into [] (-> [10, 15] (keg/rdd (map #(computeQuotUn % 5)))))

I still couldn't use clojure.data.json because the namespace dynamic variables, (i.e. value-fn, etc) are not initialized during declaration.

Thank you for looking into this.

[cgrand]

Indeed. The root of this second problem is quite different: vars are tracked only when they are set. We need to instrument the interning of vars too. You may work around the problem by doing a `(alter-var-root

'clojure.data.json/value-fn (constantly nil))` at the top of your ns.

On Fri, Jan 13, 2017 at 12:45 AM, myu07 notifications@github.com wrote:

I tested the 0.4.2 version and it didn't completely fix the problem. If the dynamic variable was initialized during declaration, then everything works fine. However, if the dynamic variable is unbound, it fails with the same exception.

(def ^{:dynamic true } divisor-un)

(defn computeQuotUn [dividend divisor] (binding [divisor-un divisor] (quot dividend divisor-un )))

(into [] (-> [10, 15] (keg/rdd (map #(computeQuotUn % 5)))))

I still couldn't use clojure.data.json because the namespace dynamic variables, (i.e. value-fn, etc) are not initialized during declaration.

Thank you for looking into this.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/HCADatalab/powderkeg/issues/9#issuecomment-272320092, or mute the thread https://github.com/notifications/unsubscribe-auth/AAC3sR2S6fMuSSKoaFWTtCsCJoIkkfL1ks5rRrszgaJpZM4LfDyg .

-- On Clojure http://clj-me.cgrand.net/ Clojure Programming http://clojurebook.com Training, Consulting & Contracting http://lambdanext.eu/

[myu07]

After invoking "alter-var-root" to 6 of clojure.data.json's dynamic variables, I finally got past this issue. Thanks for the workaround.

On another note, I found a new problem on keg but it's not related to dynamic variables. I will post a new issue.

[cgrand]

try 0.4.3-SNAPSHOT: the workaround shouldn't be needed.

[myu07]

I tested the 0.4.3-SNAPHSHOT version and it worked without using the workaround. Thank you!!!