cljdoc-analyzer is failing to analyze versions of zprint that it previously analyzed successfully

[kkinnear]

I am totally confused by a cljdoc problem I have encountered.

Presenting problem:

All of my cljdoc local trial runs, as well as cljdoc-analyzer runs, yield:

"Cannot invoke \"clojure.lang.IFn.invoke(Object, Object)\" because the return value of \"clojure.lang.Var.getRawRoot()\" is null"

It appears to successfully analyze the clj code, but fail on doing the cljs analysis (entire output available later):

 Analyzing for clj
 Clojure version 1.11.1
 ⏱  Analysis for clj took: 1.803s
 Analyzing for cljs
 ClojureScript version 1.11.60
stderr:
 {:clojure.main/message
  "Execution error (NullPointerException) at sci.impl.copy-vars/var-meta (copy_vars.cljc:65).\nCannot invoke \"clojure.lang.IFn.invoke(Object, Object)\" because the return value of \"clojure.lang.Var.getRawRoot()\" is null\n",

Background:

I routinely run cljdoc locally to ensure that my docs will work with cljdoc when I push them to GitHub. This generally works fine, and for instance I successfully ran cljdoc locally on zprint 1.2.7 prior to releasing it. It worked locally. And, indeed, when I pushed 1.2.7, cljdoc successfully analyzed my code and docs and you can see the docs for zprint 1.2.7 work fine online with cljdoc today.

I have been doing the same thing while working on zprint 1.2.8, and it was working fine. I last had it work on Sept 1. On Sept 9, I was unable to get the analysis of the code to work (received the above error), no matter what I tried. I tried a newer version of cljdoc locally, no luck. I had done very little to the code in the intervening period, so I was quite surprised. I tried a lot to things, all to no avail.

I then tried to work with older jar files, thinking I had messed something up. Still, the same problem.

I cloned the 1.2.7 version off of GitHub into a new directory, and tried that. Same problem.

I then looked at cljdoc issues. Nothing obvious. I stumbled into cljdoc-analyzer. Maybe this would help, or give better errors, or something?

For what it is worth, I'm running on macOS on MacBook Air M1, using:

clone/zprint % sdk use java 17.0.5-oracle

Using java version 17.0.5-oracle in this shell.

I installed it locally:

√ clone/zprint % clojure -Ttools install io.github.cljdoc/cljdoc-analyzer '{:git/tag "RELEASE"}' :as cljdoc
...
cljdoc: Installed io.github.cljdoc/cljdoc-analyzer RELEASE

Then I built the jar in the cloned version of zprint, 1.2.7, with

lein clean
lein install

and ran the analyzer on it:

√ clone/zprint % clojure -Tcljdoc analyze-local
2023-09-10 13:16:12,329 INFO  cljdoc-analyzer.deps-tool - args:
{:project "/Users/kkinnear/homeroot/kkinnear/clojure/clone/zprint",
 :version "snapshot",
 :pompath "pom.xml",
 :jarpath "target/zprint-1.2.7.jar"}

2023-09-10 13:16:12,331 INFO  cljdoc-analyzer.runner - cljdoc-analyzer-version 1.0.771
2023-09-10 13:16:12,340 INFO  cljdoc-analyzer.runner - args:
{:project "/Users/kkinnear/homeroot/kkinnear/clojure/clone/zprint",
 :version "snapshot",
 :jarpath "target/zprint-1.2.7.jar",
 :pompath "pom.xml",
 :extra-repos nil,
 :namespaces :all,
 :exclude-with [:no-doc :skip-wiki :mranderson/inlined],
 :output-filename
 "output-/Users/kkinnear/homeroot/kkinnear/clojure/clone/zprint-snapshot.edn"}

2023-09-10 13:16:12,613 INFO  cljdoc-analyzer.runner - dependencies for analysis:
org.clojure/clojure 1.11.1
  org.clojure/core.specs.alpha 0.2.62
  org.clojure/spec.alpha 0.3.218
zprint /private/var/folders/f6/hc_hlh0n5n1gvpvnjnnrr6200000gn/T/cljdoc--Users-kkinnear-homeroot-kkinnear-clojure-clone-zprint-snapshot4711732175783906790
  rewrite-clj/rewrite-clj 1.1.47
  org.babashka/sci 0.7.39
    borkdude/edamame 1.1.17
    org.babashka/sci.impl.types 0.0.2
    borkdude/sci.impl.reflector 0.0.1
org.clojure/clojurescript 1.11.60
  com.cognitect/transit-java 1.0.362
    com.fasterxml.jackson.core/jackson-core 2.8.7
    org.msgpack/msgpack 0.6.12
      com.googlecode.json-simple/json-simple 1.1.1
      org.javassist/javassist 3.18.1-GA
    javax.xml.bind/jaxb-api 2.3.0
  org.clojure/google-closure-library 0.0-20211011-0726fdeb
    org.clojure/google-closure-library-third-party 0.0-20211011-0726fdeb
  org.clojure/tools.reader 1.3.6
  com.google.javascript/closure-compiler-unshaded v20220502
cljdoc-analyzer/metagetta /Users/kkinnear/.gitlibs/libs/io.github.cljdoc/cljdoc-analyzer/f1b29aff3d81f01f420f414983c50d87da907616/modules/metagetta
hiccup/hiccup 1.0.5
javax.servlet/javax.servlet-api 4.0.1

2023-09-10 13:16:12,613 INFO  cljdoc-analyzer.runner - launching metagetta for: /Users/kkinnear/homeroot/kkinnear/clojure/clone/zprint languages: :auto-detect
2023-09-10 13:16:19,929 INFO  cljdoc-analyzer.runner - metagetta results:
exit-code 1
stdout:
 Args: {:namespaces :all,
        :root-path
        "/var/folders/f6/hc_hlh0n5n1gvpvnjnnrr6200000gn/T/cljdoc--Users-kkinnear-homeroot-kkinnear-clojure-clone-zprint-snapshot4711732175783906790/src/main/clojure",
        :languages :auto-detect,
        :output-filename
        "/var/folders/f6/hc_hlh0n5n1gvpvnjnnrr6200000gn/T/-Users-kkinnear-homeroot-kkinnear-clojure-clone-zprint7695835249934503793.edn",
        :exclude-with [:no-doc :skip-wiki :mranderson/inlined]}
 Java version 17.0.5
 Analyzing for clj
 Clojure version 1.11.1
 ⏱  Analysis for clj took: 1.803s
 Analyzing for cljs
 ClojureScript version 1.11.60
stderr:
 {:clojure.main/message
  "Execution error (NullPointerException) at sci.impl.copy-vars/var-meta (copy_vars.cljc:65).\nCannot invoke \"clojure.lang.IFn.invoke(Object, Object)\" because the return value of \"clojure.lang.Var.getRawRoot()\" is null\n",
  :clojure.main/triage
  {:clojure.error/class java.lang.NullPointerException,
   :clojure.error/line 65,
   :clojure.error/cause
   "Cannot invoke \"clojure.lang.IFn.invoke(Object, Object)\" because the return value of \"clojure.lang.Var.getRawRoot()\" is null",
   :clojure.error/symbol sci.impl.copy-vars/var-meta,
   :clojure.error/source "copy_vars.cljc",
   :clojure.error/phase :execution},
  :clojure.main/trace
  {:via
   [{:type clojure.lang.ExceptionInfo,
     :message
     "Could not generate ClojureScript documentation for zprint/core.cljc",
     :data {},
     :at
     [cljdoc_analyzer.metagetta.utils$default_exception_handler
      invokeStatic
      "utils.clj"
      124]}
    {:type clojure.lang.ExceptionInfo,
     :data
     {:clojure.error/source
      "file:/Users/kkinnear/.m2/repository/org/babashka/sci/0.7.39/sci-0.7.39.jar!/sci/impl/io.cljc",
      :clojure.error/line 55,
      :clojure.error/column 17,
      :clojure.error/phase :macroexpansion,
      :clojure.error/symbol sci.impl.copy-vars/copy-var},
     :at
     [cljs.analyzer$macroexpand_1_STAR_$fn__3704
      invoke
      "analyzer.cljc"
      4029]}
    {:type java.lang.NullPointerException,
     :message
     "Cannot invoke \"clojure.lang.IFn.invoke(Object, Object)\" because the return value of \"clojure.lang.Var.getRawRoot()\" is null",
     :at
     [sci.impl.copy_vars$var_meta invokeStatic "copy_vars.cljc" 65]}],
...

and I get the same problem. When I look at my local cljdoc output for zprint 1.2.7, I see:

[API import failed in build #47](http://localhost:8000/builds/47)

but when I look at the "real" cljdoc online, for zprint 1.2.7, I see that it works fine.

I've attached the file log.txt which contains the entirety of the output from the above command: log.txt.

I am totally confused by this problem, and have no idea what could be wrong. Given that I successfully analyzed zprint 1.2.7 with my local copy of cljdoc running (where I run the sever locally), and now cannot successfully analyze that same code, I'm at a loss as to what could be the problem. I had upgraded babashka along the way, but tried downgrading it to a previous version -- no change.

I've probably done something that is causing this, but I have absolutely no idea what it could be. Any help would be appreciated.

Thanks!

[lread]

Hi @kkinnear, thanks for the detailed report, I shall take a peek!

[lread]

Can reproduce your issue here with the current version of cljdoc-analyzer. Cljdoc-analyzer v1.0.759 (the previous version) analyzes zprint 1.2.7 just fine.

I'll dig into what is going on.

[lread]

Hmmm... still digging. A change I made delays the require of ClojureScript analysis support to when/if needed. This is somehow affecting sci workings under ClojureScript analysis. I don't understand why yet. If I revert this delayed require analysis works.

Both sci and cljdoc-analyzer interact with the Clojurescript analyzer api... so... probably related somehow to that.. dunno yet... I'll keep looking.

[lread]

I don't understand why, but if I move my delayed require before any analysis begins, the analysis succeeds. I'll do that for now and dig deeper someday if necessary.

[lread]

@kkinnear I've fixed the issue in cljdoc-analyzer, I'll ping back here when it has gone live for cljdoc.

[lread]

@kkinnear the fix is now live!

[kkinnear]

Wow! @., that’s amazing. Thank you so much. I didn’t know that the local copy of cljdoc I have would update itself as new things were released. I hope to have time give it a spin tomorrow!Thanks again!On Sep 10, 2023, at 6:06 PM, Lee Read @.> wrote: @kkinnear the fix is now live!

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[lread]

For you @kkinnear? My pleasure!

The cljdoc server is set up to always use the current release of cljdoc-analyzer.

[kkinnear]

Thanks again, @lread, for the extremely quick fix. It works great! I didn't know you had been working on cljdoc. It is a great service. It also uses zprint :-).

I'm guessing that this wasn't a common problem, but more likely (almost?) unique to zprint. If so, I wouldn't be surprised if the problem was related to the fact that zprint uses sci internally, but that's just a guess.

As an aside, I saw the issue for cljdoc about analyzing library API's statically, #543. I think zprint would be one of the ones that would work statically, for what that's worth.

Now I'm back in business trying to finish up the 1.2.8 release of zprint. Because @borkdude opened up some sci code in babashka recently, zprint 1.2.8 is now babashka compatible. I've documented how people can run zprint as a task in babashka or install it with bbin.

Anyway, thanks again! It is always great to work with you.

[lread]

Thanks again, @lread, for the extremely quick fix. It works great! I didn't know you had been working on cljdoc. It is a great service. It also uses zprint :-).

Yeah, I help out here and there on cljdoc! And zprint is a great choice for cljdoc to format edn presented to the user!

I'm guessing that this wasn't a common problem, but more likely (almost?) unique to zprint. If so, I wouldn't be surprised if the problem was related to the fact that zprint uses sci internally, but that's just a guess.

It was somehow sci related but not sure how/why. Both sci and cljdoc-analyzer use the ClojureScript analyzer, so some order of init/operations issue there somehow, I guess.

As an aside, I saw the issue for cljdoc about analyzing library API's statically, #543. I think zprint would be one of the ones that would work statically, for what that's worth.

Thanks for the tip, I've been slowly working on this over the long past while. I'll be sure to add zprint to my exploration suite.

Now I'm back in business trying to finish up the 1.2.8 release of zprint. Because @borkdude opened up some sci code in babashka recently, zprint 1.2.8 is now babashka compatible. I've documented how people can run zprint as a task in babashka or install it with bbin.

Cool!

Anyway, thanks again! It is always great to work with you.

The feeling is mutual!