StackOverflow while compiling meander expression

[timothypratley]

meanderbug.zip

Attached is a project, if you run lein check it fails to compile with a StackOverflow while expanding the meander macro. If you send it to the REPL, it compiles just fine... weird!!!

=== UPDATE ===

You can work around this issue by increasing the JVM stack size, which is quite small by default.

Add :jvm-opts ["-Xss2m"] to your project.clj (at the top level) to allocate a larger (2megabyte) stack.

[timothypratley]

There doesn't appear to be anything special about the form that fails to compile:

(ns meanderbug.core
  (:require [meander.epsilon :as m]))

(defn z []
  (m/rewrite {}
    (loop* [?seq (clojure.core/seq ?xs)
            ?chunk nil
            ?chunkn 0
            ?i 0]
      (if (clojure.lang.Numbers/lt ?i ?chunkn)
        (let* [(m/and ?sym (m/app always-meta {:tag  ?tag
                                               :type ?type})) (.nth ?chunk ?i)]
          (do ?body (recur ?seq ?chunk ?chunkn (clojure.lang.Numbers/unchecked_inc ?i))))
        (let* [?as (clojure.core/seq ?seq)]
          (if ?as
            (let* [?bs ?as]
              (if (clojure.core/chunked-seq? ?bs)
                (let* [?cs (clojure.core/chunk-first ?bs)]
                  (recur
                    (clojure.core/chunk-rest ?bs)
                    ?cs
                    (clojure.lang.RT/intCast (clojure.lang.RT/count ?cs))
                    (clojure.lang.RT/intCast 0)))
                (let* [?sym (clojure.core/first ?bs)]
                  (do ?body (recur (clojure.core/next ?bs) nil 0 0)))))))))
    (foreach ~(or ?type ?tag) ?sym ?xs (m/app inner-form ?body))))

I've tried cutting out parts of the form and one weird thing is that if I remove either sub-expressions, it compiles! This seems odd to me because it is not a particularly deeply nested structure.

[timothypratley]

@noprompt any debugging suggestions? I'd like to try to narrow this issue down but not sure where to start.

[noprompt]

@timothypratley Not sure. Here's what I see:

> lein check
Compiling namespace meanderbug.core
Performance warning, meander/util/epsilon.cljc:85:3 - case has int tests, but tested expression is not primitive.
Performance warning, meander/util/epsilon.cljc:119:3 - case has int tests, but tested expression is not primitive.
Performance warning, meander/util/epsilon.cljc:153:3 - case has int tests, but tested expression is not primitive.
Performance warning, meander/util/epsilon.cljc:183:3 - case has int tests, but tested expression is not primitive.
Performance warning, meander/util/epsilon.cljc:276:6 - case has int tests, but tested expression is not primitive.
Performance warning, meander/util/epsilon.cljc:253:4 - case has int tests, but tested expression is not primitive.
Performance warning, meander/util/epsilon.cljc:337:6 - case has int tests, but tested expression is not primitive.
Performance warning, meander/util/epsilon.cljc:311:4 - case has int tests, but tested expression is not primitive.
Reflection warning, meander/util/epsilon.cljc:404:20 - reference to field length can't be resolved.
Reflection warning, meander/util/epsilon.cljc:417:18 - reference to field length can't be resolved.
Performance warning, meander/util/epsilon.cljc:397:3 - case has int tests, but tested expression is not primitive.
Performance warning, meander/util/epsilon.cljc:551:3 - case has int tests, but tested expression is not primitive.
Reflection warning, meander/util/epsilon.cljc:616:21 - reference to field val can't be resolved.
Performance warning, meander/match/syntax/epsilon.cljc:687:3 - case has int tests, but tested expression is not primitive.
Performance warning, meander/match/syntax/epsilon.cljc:816:3 - case has int tests, but tested expression is not primitive.
Reflection warning, meander/match/runtime/epsilon.cljc:172:34 - reference to field length can't be resolved.
Reflection warning, meander/match/runtime/epsilon.cljc:172:14 - call to method slice can't be resolved (target class is unknown).
Reflection warning, meander/match/runtime/epsilon.cljc:177:20 - call to method slice can't be resolved (target class is unknown).
Reflection warning, meander/match/runtime/epsilon.cljc:185:12 - call to method slice can't be resolved (target class is unknown).
Reflection warning, meander/match/runtime/epsilon.cljc:191:38 - call to method slice can't be resolved (target class is unknown).
Reflection warning, meander/match/runtime/epsilon.cljc:201:12 - call to method slice can't be resolved (target class is unknown).
Reflection warning, meander/match/runtime/epsilon.cljc:203:16 - call to method slice can't be resolved (target class is unknown).
Reflection warning, meander/matrix/epsilon.cljc:295:26 - call to method indexOf can't be resolved (target class is unknown).
Reflection warning, meander/matrix/epsilon.cljc:299:26 - call to method indexOf can't be resolved (target class is unknown).
Reflection warning, meander/match/epsilon.cljc:247:11 - call to method indexOf can't be resolved (target class is unknown).
Performance warning, meander/match/epsilon.cljc:783:7 - case has int tests, but tested expression is not primitive.

---

Recently, someone at work encountered a similar situation to the one you're in and IIRC it seemed possibly related to clojure.tools.namespace. Does your profile.clj have dependencies in it? If so, try seeing if any of them may be interfering.

[timothypratley]

I tried removing my ~/.lein/profile and still see the behavior. I see the same output you posted, but additionally:

Unexpected error (StackOverflowError) macroexpanding m/rewrite at (meanderbug/core.clj:5:3).

and stack trace attached:

clojure-2077442374618026138.txt

[timothypratley]

@noprompt FYI I've been able to narrow this down a bit! Adding :jvm-opts ["-Xss512k"] should produce the stack overflow for you, can you confirm? And :jvm-opts ["-Xss1m"] should be fine for lein check I think this means that due to environments our default stack sizes are different... and as a work around I can certainly just set a bigger stack size... but I think it also will be something people bump into and I wonder if the stack consumption could be reduced or if there is a fundamental reason why it needs to be bigger.

[timothypratley]

Just for reference here's the same pattern with all the symbols replaced with a 1 that exhibits the behavior with the stack size:

(defn z []
  (m/rewrite {}
    (1 [?seq (1 ?xs)
            ?chunk 1
            ?chunkn 1
            ?i 1]
      (1 (1 ?i ?chunkn)
        (1 [(1 ?sym (1 1 {1 ?tag
                          2 ?type}))
            (1 ?chunk ?i)]
          (1 ?body (1 ?seq ?chunk ?chunkn (1 ?i))))
        (1 [?as (1 ?seq)]
          (1 ?as
            (1 [?bs ?as]
              (1 (1 ?bs)
                (1 [?cs (1 ?bs)]
                  (1
                    (1 ?bs)
                    ?cs
                    (1 (1 ?cs))
                    (1 1)))
                (1 [?sym (1 ?bs)]
                  (1 ?body (1 (1 ?bs) 1 1 1)))))))))
    (1 ~(or ?type ?tag) ?sym ?xs (1 1 ?body))))

The example in the zip is distracting because of all the symbols (sorry about that), so this version makes it clear it's just a nested structure with some logic variables.

[noprompt]

I was able to reproduce the error with :jvm-opts ["-Xss512k"]

I think it also will be something people bump into

I agree and, actually, bumping up the stack size ended up being the prescription for the person I mentioned earlier.

I wonder if the stack consumption could be reduced

I'm certain it could be.

Pending a thorough investigation, perhaps we should update the README, etc.

[timothypratley]

👍 I'll submit a PR