search

cljdoc / cljdoc-analyzer

Get resolved metadata from Clojure and ClojureScript projects
Eclipse Public License 2.0
9 stars 7 forks source link

Cljdoc-analyzer can fail mysteriously #34

Open lread opened 3 years ago

lread commented 3 years ago

This issue originated from a diagnosis session on Slack with @mccraigmccraig.

The ingestion of alia on cljdoc was failing with less than helpful error messaging.

Here's what was showing on circleci:

WARNING: When invoking clojure.main, use -M
2020-12-22 18:10:17,140 INFO  cljdoc-analyzer.cljdoc-main - args:
{:project "cc.qbits/alia",
 :version "5.0.0-alpha1",
 :jarpath
 "https://repo.clojars.org/cc/qbits/alia/5.0.0-alpha1/alia-5.0.0-alpha1.jar",
 :pompath
 "https://repo.clojars.org/cc/qbits/alia/5.0.0-alpha1/alia-5.0.0-alpha1.pom",
 :repos
 {"clojars" {:url "https://repo.clojars.org/"},
  "central" {:url "https://repo.maven.apache.org/maven2/"}}}

2020-12-22 18:10:17,155 INFO  cljdoc-analyzer.runner - args:
{:project "cc.qbits/alia",
 :version "5.0.0-alpha1",
 :jarpath
 "https://repo.clojars.org/cc/qbits/alia/5.0.0-alpha1/alia-5.0.0-alpha1.jar",
 :pompath
 "https://repo.clojars.org/cc/qbits/alia/5.0.0-alpha1/alia-5.0.0-alpha1.pom",
 :extra-repos nil,
 :namespaces :all,
 :exclude-with [:no-doc :skip-wiki],
 :output-filename
 "/tmp/cljdoc/analysis-out/cljdoc-edn/cc.qbits/alia/5.0.0-alpha1/cljdoc.edn"}

2020-12-22 18:10:17,160 INFO  cljdoc-analyzer.runner - Downloading https://repo.clojars.org/cc/qbits/alia/5.0.0-alpha1/alia-5.0.0-alpha1.jar
2020-12-22 18:10:17,722 ERROR cljdoc-analyzer.runner - nil
2020-12-22 18:10:17,722 ERROR cljdoc-analyzer.runner - STDOUT
 nil
2020-12-22 18:10:17,722 ERROR cljdoc-analyzer.runner - STDERR
 nil

We managed to reproduce this failure locally and after some trial and error we noticed that alia's lein project was generating a pom with dependencyManagement that included org.clojure/clojure. The dependencies section later included, as you might expect, org.clojure/clojure without any specific version.

Here's a minimal project.clj to illustrate. I've included an additional arbritray dependency to demonstrate a point.

(defproject lread/repro-odd-cljdoc-failure "1.0.0"
  :description "Trying to reproduce cljdoc failure"

  ;; it seems to be caused by include clojure as a managed dependency
  :managed-dependencies [[org.clojure/clojure "1.10.1"]
                         [version-clj/version-clj "0.1.2"]]

  ;; and referencing it without a version
  :dependencies [[org.clojure/clojure]
                 [version-clj/version-clj]])

We'll include a dummy namespace under src/my_stuff/api.clj just so we have something to analyze:

(ns my-stuff.api
  (:require [version-clj.core :refer [version-compare]]))

(defn oh-hi []
  ;; just referencing a lib for the sake of referncing a lib
  (version-compare "1.0.0" "1.0.0"))

Steps to reproduce:

1. install locally lein install

2. run cljdoc-analyzer I ran from source:

clojure -m cljdoc-analyzer.main analyze \
  --project  lread/repro-odd-cljdoc-failure --version 1.0.0 \
  --output-filename "/tmp/blarp.edn" \
  --exclude-with :no-doc \
  --exclude-with :skip-wiki

result

WARNING: When invoking clojure.main, use -M
2020-12-23 15:03:56,682 INFO  cljdoc-analyzer.runner - args:
{:project "lread/repro-odd-cljdoc-failure",
 :version "1.0.0",
 :exclude-with [:no-doc :skip-wiki],
 :output-filename "/tmp/erp.edn",
 :jarpath
 "/Users/lee/.m2/repository/lread/repro-odd-cljdoc-failure/1.0.0/repro-odd-cljdoc-failure-1.0.0.jar",
 :pompath
 "/Users/lee/.m2/repository/lread/repro-odd-cljdoc-failure/1.0.0/repro-odd-cljdoc-failure-1.0.0.pom",
 :extra-repos {}}

2020-12-23 15:03:56,800 ERROR cljdoc-analyzer.runner - nil
2020-12-23 15:03:56,800 ERROR cljdoc-analyzer.runner - STDOUT
 nil
2020-12-23 15:03:56,800 ERROR cljdoc-analyzer.runner - STDERR
 nil

work-around If we put explicitly specified a version for org.clojure/clojure under dependencies things analyze just fine.

  :dependencies [[org.clojure/clojure "1.10.1"]
                 [version-clj/version-clj]])

And rerun lein install Followed by cljdoc-analyzer (see step 2 above). Then the analysis succeeds. Notice that version-clj/version-clj could remain maven version free.

thoughts We don't know why the ingest failed, but we do know:

We can at least report the symptom, which I suspect might be "no dependencies found".

lread commented 3 years ago

Same symptom has occurred for https://github.com/logicblocks/salutem for @tobyclemson.

If I change cljdoc-analyzer to log full exception, for salutem I see:

2021-10-30 21:22:43,860 ERROR cljdoc-analyzer.runner - Unable to analyze
java.lang.NullPointerException: null
    at java.base/java.util.regex.Matcher.getTextLength(Matcher.java:1770)
    at java.base/java.util.regex.Matcher.reset(Matcher.java:416)
    at java.base/java.util.regex.Matcher.<init>(Matcher.java:253)
    at java.base/java.util.regex.Pattern.matcher(Pattern.java:1133)
    at clojure.core$re_matcher.invokeStatic(core.clj:4845)
    at clojure.core$re_matcher.invoke(core.clj:4838)
    at version_clj.split$split_once.invokeStatic(split.cljc:24)
    at version_clj.split$split_once.invoke(split.cljc:21)
    at version_clj.split$split_known_qualifier.invokeStatic(split.cljc:116)
    at version_clj.split$split_known_qualifier.invoke(split.cljc:113)
    at version_clj.split$split_version_and_qualifier.invokeStatic(split.cljc:127)
    at version_clj.split$split_version_and_qualifier.invoke(split.cljc:123)
    at version_clj.split$version__GT_seq.invokeStatic(split.cljc:142)
    at version_clj.split$version__GT_seq.doInvoke(split.cljc:139)
    at clojure.lang.RestFn.invoke(RestFn.java:410)
    at version_clj.compare$version_compare.invokeStatic(compare.cljc:95)
    at version_clj.compare$version_compare.doInvoke(compare.cljc:90)
    at clojure.lang.RestFn.invoke(RestFn.java:442)
    at version_clj.core$version_compare.invokeStatic(core.cljc:28)
    at version_clj.core$version_compare.doInvoke(core.cljc:23)
    at clojure.lang.RestFn.invoke(RestFn.java:425)
    at cljdoc_analyzer.deps$ensure_recent_ish$choose_version__12671.invoke(deps.clj:18)
    at clojure.lang.AFn.applyToHelper(AFn.java:156)
    at clojure.lang.AFn.applyTo(AFn.java:144)
    at clojure.core$apply.invokeStatic(core.clj:669)
    at clojure.core$update_in$up__6870.invoke(core.clj:6174)
    at clojure.core$update_in$up__6870.invoke(core.clj:6173)
    at clojure.core$update_in.invokeStatic(core.clj:6175)
    at clojure.core$update_in.doInvoke(core.clj:6161)
    at clojure.lang.RestFn.invoke(RestFn.java:467)
    at cljdoc_analyzer.deps$ensure_recent_ish$fn__12674.invoke(deps.clj:21)
    at clojure.core.protocols$iter_reduce.invokeStatic(protocols.clj:49)
    at clojure.core.protocols$fn__8162.invokeStatic(protocols.clj:75)
    at clojure.core.protocols$fn__8162.invoke(protocols.clj:75)
    at clojure.core.protocols$fn__8110$G__8105__8123.invoke(protocols.clj:13)
    at clojure.core$reduce.invokeStatic(core.clj:6830)
    at clojure.core$reduce.invoke(core.clj:6812)
    at cljdoc_analyzer.deps$ensure_recent_ish.invokeStatic(deps.clj:19)
    at cljdoc_analyzer.deps$ensure_recent_ish.invoke(deps.clj:10)
    at cljdoc_analyzer.deps$deps.invokeStatic(deps.clj:94)
    at cljdoc_analyzer.deps$deps.invoke(deps.clj:82)
    at cljdoc_analyzer.deps$resolved_deps.invokeStatic(deps.clj:106)
    at cljdoc_analyzer.deps$resolved_deps.invoke(deps.clj:97)
    at cljdoc_analyzer.runner$get_metadata_STAR_.invokeStatic(runner.clj:178)
    at cljdoc_analyzer.runner$get_metadata_STAR_.invoke(runner.clj:165)
    at cljdoc_analyzer.runner$get_metadata.invokeStatic(runner.clj:211)
    at cljdoc_analyzer.runner$get_metadata.invoke(runner.clj:198)
    at cljdoc_analyzer.runner$analyze_BANG_.invokeStatic(runner.clj:233)
    at cljdoc_analyzer.runner$analyze_BANG_.invoke(runner.clj:216)
    at cljdoc_analyzer.main$analyze.invokeStatic(main.clj:21)
    at cljdoc_analyzer.main$analyze.invoke(main.clj:17)
    at cli_matic.core$invoke_subcmd.invokeStatic(core.cljc:546)
    at cli_matic.core$invoke_subcmd.invoke(core.cljc:525)
    at cli_matic.core$run_cmd_STAR_.invokeStatic(core.cljc:589)
    at cli_matic.core$run_cmd_STAR_.invoke(core.cljc:560)
    at cli_matic.core$run_cmd.invokeStatic(core.cljc:601)
    at cli_matic.core$run_cmd.invoke(core.cljc:591)
    at cljdoc_analyzer.main$_main.invokeStatic(main.clj:60)
    at cljdoc_analyzer.main$_main.doInvoke(main.clj:59)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.lang.Var.applyTo(Var.java:705)
    at clojure.core$apply.invokeStatic(core.clj:667)
    at clojure.main$main_opt.invokeStatic(main.clj:514)
    at clojure.main$main_opt.invoke(main.clj:510)
    at clojure.main$main.invokeStatic(main.clj:664)
    at clojure.main$main.doInvoke(main.clj:616)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.lang.Var.applyTo(Var.java:705)
    at clojure.main.main(main.java:40)

And this seems like cljdoc-analyzer having no concept of a pom's <depencyManagement> section. It just grabs the version-less dependency and does at test against that for a minimum version.

So two things:

  1. Why shouldn't cljdoc-analyzer understand <depencyManagement>? There might be a good reason, I just don't remember any.
  2. Why not log full exception on error? Were we worried about leaking maybe a secret or something?
SevereOverfl0w commented 2 years ago

I don't remember how this bit of code works, but I do remember thinking that poking at the POM directly was a recipe for trouble.

There are some Apache Maven APIs for dealing with POMs, maybe those can give you a "resolved version" or something.

lread commented 2 years ago

Thanks @SevereOverfl0w, yeah, I think you are right. We are already using clojure tools deps, it probably already can give us what cljdoc-analyzer needs to know. Maybe. And I guess we should question if cljdoc-analyzer really needs to know what it thinks it needs to know here.