search

nasser / magic

Morgan And Grand Iron Clojure
http://nas.sr/magic/
365 stars 18 forks source link

`walk/postwalk` with `datafy` cause error #226

Closed skydread1 closed 1 year ago

skydread1 commented 2 years ago

Problem

In case the optimisation flag magic.flags/*strongly-typed-invokes* is true, clojure.datafy/datafy for nested records throws an exception when used with clojure.walk/postwalk.

How to reproduce

Let's consider the Clojure code:


(:require [clojure.core.protocols :as proto]
          [clojure.datafy :refer [datafy]]
          [clojure.walk :as walk])

(defrecord R1 [a]
  proto/Datafiable
  (datafy [_]
    (datafy a)))

(defrecord R2 [r1]
  proto/Datafiable
  (datafy [_]
    {:r1 (datafy r1)}))

(defrecord R3 [r1]
  proto/Datafiable
  (datafy [_]
    {:r1 (walk/postwalk datafy r1}))

(def r (->R2 (->R1 1)))

(def r2 (->R3 [(->R1 1) (->R1 2)]))

In both JVM and CLR, we get the expected result for:

;; clojure or nostrand repl
(datafy r)
=> {:r1 1}

However, calling datafy on r2 triggers an error on the CLR:

;; clojure repl
(datafy r2)
=> {:r1 [1 2]}

;; nostrand repl
(datafy r2)
Exception System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidCastException: Specified cast is not valid.
  at user.R1.clojure.lang.IPersistentCollection.cons (System.Object ) [0x0000f] in <c6181a0eda7645dc89438b81528e4348>:0 
  at clojure.lang.RT.conj (clojure.lang.IPersistentCollection coll, System.Object x) [0x00012] in <4c496cda35d84170b4a3d28ddac68bdb>:0 
  at <magic>clojure_core$conj__0.invokeTyped (System.Object coll, System.Object x) [0x00000] in <732fe409070640e085953f1349ea313b>:0 
  at <magic>clojure_core$conj__0.invoke (System.Object coll, System.Object x) [0x00000] in <732fe409070640e085953f1349ea313b>:0 
  at <magic>clojure_walk$<fn>__0.invoke (System.Object r, System.Object x) [0x00021] in <aa39d73e465e4b5184bac11b604d7883>:0 
  at clojure.lang.AFn.ApplyToHelper (clojure.lang.IFn fn, clojure.lang.ISeq argList) [0x000b3] in <4c496cda35d84170b4a3d28ddac68bdb>:0 
  at clojure.lang.AFn.applyTo (clojure.lang.ISeq arglist) [0x0000c] in <4c496cda35d84170b4a3d28ddac68bdb>:0 
  at clojure.lang.AFunction+MetaWrapper.doInvoke (System.Object args) [0x00001] in <4c496cda35d84170b4a3d28ddac68bdb>:0 
  at clojure.lang.RestFn.invoke (System.Object arg1, System.Object arg2) [0x00044] in <4c496cda35d84170b4a3d28ddac68bdb>:0 
  at <magic>clojure_core_protocols$iter-reduce__0.invokeTyped (System.Collections.IEnumerable coll, System.Object f, System.Object val) [0x00027] in <5bafdf61553f4056bf99c888a19932bc>:0 
  at <magic>clojure_core_protocols$iter-reduce__0.invoke (System.Object coll, System.Object f, System.Object val) [0x00000] in <5bafdf61553f4056bf99c888a19932bc>:0 
  at <magic>clojure_core_protocols$CollReduce-System_Collections_IEnumerable17705__0.invokeTyped (System.Collections.IEnumerable coll, System.Object f, System.Object val) [0x0000a] in <5bafdf61553f4056bf99c888a19932bc>:0 
  at <magic>clojure_core_protocols$CollReduce-System_Collections_IEnumerable17705__0.invoke (System.Object coll, System.Object f, System.Object val) [0x00000] in <5bafdf61553f4056bf99c888a19932bc>:0 
  at <magic>clojure_core_protocols$G__17527__0.invoke (System.Object gf__coll__17539, System.Object gf__f__17540, System.Object gf__val__17541) [0x0004a] in <5bafdf61553f4056bf99c888a19932bc>:0 
  at <magic>clojure_core$reduce__0.invoke (System.Object f, System.Object val, System.Object coll) [0x00018] in <732fe409070640e085953f1349ea313b>:0 
  at <magic>clojure_walk$walk__0.invoke (System.Object inner, System.Object outer, System.Object form) [0x001f8] in <aa39d73e465e4b5184bac11b604d7883>:0 
  at <magic>clojure_walk$postwalk__0.invoke (System.Object f, System.Object form) [0x00025] in <aa39d73e465e4b5184bac11b604d7883>:0 
  at <magic>clojure_core$<fn>__27.invoke (System.Object x) [0x00000] in <732fe409070640e085953f1349ea313b>:0 
  at clojure.lang.AFn.ApplyToHelper (clojure.lang.IFn fn, clojure.lang.ISeq argList) [0x0008a] in <4c496cda35d84170b4a3d28ddac68bdb>:0 
  at clojure.lang.RestFn.applyTo (clojure.lang.ISeq arglist) [0x00024] in <4c496cda35d84170b4a3d28ddac68bdb>:0 
  at clojure.lang.AFunction+MetaWrapper.doInvoke (System.Object args) [0x00001] in <4c496cda35d84170b4a3d28ddac68bdb>:0 
  at clojure.lang.RestFn.invoke (System.Object arg1) [0x0002e] in <4c496cda35d84170b4a3d28ddac68bdb>:0 
  at <magic>clojure_core$<fn>__34.invoke () [0x00163] in <732fe409070640e085953f1349ea313b>:0 
  at clojure.lang.LazySeq.sval () [0x0000f] in <4c496cda35d84170b4a3d28ddac68bdb>:0 
  at (wrapper synchronized) clojure.lang.LazySeq.sval()
  at clojure.lang.LazySeq.seq () [0x00001] in <4c496cda35d84170b4a3d28ddac68bdb>:0 
  at (wrapper synchronized) clojure.lang.LazySeq.seq()
  at clojure.lang.RT.seq (System.Object coll) [0x00025] in <4c496cda35d84170b4a3d28ddac68bdb>:0 
  at <magic>clojure_core$seq__0.invokeTyped (System.Object coll) [0x00000] in <732fe409070640e085953f1349ea313b>:0 
  at <magic>clojure_core$seq__0.invoke (System.Object coll) [0x00000] in <732fe409070640e085953f1349ea313b>:0 
  at <magic>clojure_core_protocols$seq-reduce__0.invoke (System.Object coll, System.Object f, System.Object val) [0x0000a] in <5bafdf61553f4056bf99c888a19932bc>:0 
  at <magic>clojure_core_protocols$CollReduce-clojure_lang_LazySeq17699__0.invokeTyped (clojure.lang.LazySeq coll, System.Object f, System.Object val) [0x0000a] in <5bafdf61553f4056bf99c888a19932bc>:0 
  at <magic>clojure_core_protocols$CollReduce-clojure_lang_LazySeq17699__0.invoke (System.Object coll, System.Object f, System.Object val) [0x00000] in <5bafdf61553f4056bf99c888a19932bc>:0 
  at <magic>clojure_core_protocols$G__17527__0.invoke (System.Object gf__coll__17539, System.Object gf__f__17540, System.Object gf__val__17541) [0x0005c] in <5bafdf61553f4056bf99c888a19932bc>:0 
  at <magic>clojure_core$reduce__0.invoke (System.Object f, System.Object val, System.Object coll) [0x00018] in <732fe409070640e085953f1349ea313b>:0 
  at <magic>clojure_core$into__0.invoke (System.Object to, System.Object from) [0x0007f] in <732fe409070640e085953f1349ea313b>:0 
  at <magic>clojure_walk$walk__0.invoke (System.Object inner, System.Object outer, System.Object form) [0x00111] in <aa39d73e465e4b5184bac11b604d7883>:0 
  at <magic>clojure_walk$postwalk__0.invoke (System.Object f, System.Object form) [0x00025] in <aa39d73e465e4b5184bac11b604d7883>:0 
  at user.R3.datafy () [0x00044] in <e1bbf028ca3c427cbf51ccc50c3af52f>:0 
  at <magic>clojure_core_protocols$<fn>__6.invokeTyped (clojure.core.protocols.Datafiable gf__o__17914) [0x00000] in <5bafdf61553f4056bf99c888a19932bc>:0 
  at <magic>clojure_core_protocols$<fn>__6.invoke (System.Object gf__o__17914) [0x00000] in <5bafdf61553f4056bf99c888a19932bc>:0 
  at <magic>clojure_core_protocols$G__17911__0.invoke (System.Object gf__o__17916) [0x0004a] in <5bafdf61553f4056bf99c888a19932bc>:0 
  at <magic>clojure_datafy$datafy__0.invoke (System.Object x) [0x0000a] in <816129cc58594472b393e5ab1d3af8f1>:0 
  at <magic>expr_123.eval () [0x0002d] in <fdb82b42e7bd40ff84efd801687d6c7e>:0 
  at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
  at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <e068e2227ab74c1bb3d724ebaab0e3ff>:0 
   --- End of inner exception stack trace ---
  at Magic.Dispatch.InvokeUnwrappingExceptions (System.Reflection.MethodBase method, System.Object target, System.Object[] args) [0x00015] in <79947ecf2ac04d5b8a37881fb4867092>:0 
  at Magic.Dispatch.InvokeInstanceMethod (System.Object o, System.String name, System.Object[] args) [0x0001c] in <79947ecf2ac04d5b8a37881fb4867092>:0 
  at <magic>magic_api$compile-expression__0.invoke (System.Object expr, System.Object ctx, System.Object opts) [0x0032f] in <c04f41b8680e46849894d0a398cadce0>:0 
  at <magic>magic_api$compile-expression-top-level__0.invoke (System.Object expr, System.Object ctx, System.Object opts) [0x000e3] in <c04f41b8680e46849894d0a398cadce0>:0 
  at <magic>magic_api$eval__0.invoke (System.Object expr) [0x000d9] in <c04f41b8680e46849894d0a398cadce0>:0 
  at clojure.lang.AFn.ApplyToHelper (clojure.lang.IFn fn, clojure.lang.ISeq argList) [0x0008a] in <4c496cda35d84170b4a3d28ddac68bdb>:0 
  at clojure.lang.AFn.applyTo (clojure.lang.ISeq arglist) [0x0000c] in <4c496cda35d84170b4a3d28ddac68bdb>:0 
  at clojure.lang.AFunction+MetaWrapper.doInvoke (System.Object args) [0x00001] in <4c496cda35d84170b4a3d28ddac68bdb>:0 
  at clojure.lang.RestFn.invoke (System.Object arg1) [0x0002e] in <4c496cda35d84170b4a3d28ddac68bdb>:0 
  at clojure.lang.Var.invoke (System.Object arg1) [0x00011] in <4c496cda35d84170b4a3d28ddac68bdb>:0 
  at <magic>clojure_core$eval__0.invoke (System.Object form) [0x0000a] in <732fe409070640e085953f1349ea313b>:0 
  at <magic>nostrand_repl$cli__0.invoke (System.Object p__1499) [0x0020c] in <3f04acc075cc4f85a04ba9fe03cef3ab>:0

So the error comes from (walk/postwalk datafy r1) when r1 is a vector of records.

skydread1 commented 2 years ago

Attempt

I notice we don't have this line of code in stdlib/walk and I tried to update it and recompile but got the same result.

nasser commented 2 years ago

From our notes

There is no datafy related error if we do not have optimisation on. The *strongly-typed-invokes* flags introduced errors concerning datafy. Also, we have less datafy errors if we type hint the code.

nasser commented 2 years ago

Reduced to a more minimal reproducing case.

(require  '[clojure.core.protocols :as proto]
          '[clojure.datafy :refer [datafy]]
          '[clojure.walk :as walk])

(defrecord R1 [a]
  proto/Datafiable
  (datafy [_]
    (datafy a)))

(walk/postwalk datafy (->R1 1))

This behaves as you described, fine without *strongly-typed-invokes* enabled and throws when it is. Likely that MAGIC's more strict notion of type casting is clashing with Clojure's protocol mechanism.

nasser commented 1 year ago

Fixed with 768f6bb9f74a795bbefab43e5d56814e82caa544.