clj.native-image doesn't use the constructed classpath

[lvh]

When additional aliases add extra deps/paths and are added to the classpath, they are not included in the classpath clj.native-image uses. I've added a demo here: https://github.com/lvh/jdnsmith

deps.edn:

{:deps {org.clojure/clojure {:mvn/version "1.10.0"}
        org.clojure/data.json {:mvn/version "0.2.6"}}
 :aliases
 {:srctwo
  {:extra-deps {camel-snake-kebab {:mvn/version "0.4.0"}}
   :extra-paths ["srctwo/"]}

  :native-image
  {:main-opts ["-m clj.native-image jdnsmith.core"
               "--report-unsupported-elements-at-runtime"
               "--initialize-at-build-time"
               "-H:Name=json2edn"]
   :jvm-opts ["-Dclojure.compiler.direct-linking=true"]
   :extra-deps
   {clj.native-image
    {:git/url "https://github.com/taylorwood/clj.native-image.git"
     :sha "567176ddb0f7507c8b0969e0a10f60f848afaf7d"}}}

  :uberdeps
  {:extra-deps {uberdeps {:mvn/version "0.1.3"}}
   :main-opts ["-m" "uberdeps.uberjar"]}

  :pack
  {:extra-deps
   {pack/pack.alpha {:git/url "https://github.com/juxt/pack.alpha.git"
                     :sha "81b9e47d992b17aa3e3af1a47aed1f0287ebe9b8"}}
   :main-opts ["-m"]}

  :depstar
  {:extra-deps
   {seancorfield/depstar {:mvn/version "0.2.1"}}}}}

When running native-image, only the toplevel :deps/:paths are used, even if the srctwo alias is active:

jdnsmith ➤ clj -A:srctwo:native-image                                                                                                                                                                                                                                                                                                           git:master
Compiling jdnsmith.core
Building native image with classpath 'classes:src:/home/lvh/.m2/repository/org/clojure/clojure/1.10.0/clojure-1.10.0.jar:/home/lvh/.m2/repository/org/clojure/data.json/0.2.6/data.json-0.2.6.jar:/home/lvh/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar:/home/lvh/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar'
[json2edn:39963]    classlist:   3,244.86 ms
[json2edn:39963]        (cap):     807.98 ms
[json2edn:39963]        setup:   1,906.19 ms
[json2edn:39963]   (typeflow):   5,416.23 ms
[json2edn:39963]    (objects):   2,571.23 ms
[json2edn:39963]   (features):     168.06 ms
[json2edn:39963]     analysis:   8,290.84 ms
[json2edn:39963]     (clinit):     143.40 ms
[json2edn:39963]     universe:     393.45 ms
[json2edn:39963]      (parse):     323.02 ms
[json2edn:39963]     (inline):   1,342.40 ms
[json2edn:39963]    (compile):   3,429.39 ms
[json2edn:39963]      compile:   5,508.10 ms
[json2edn:39963]        image:     711.84 ms
[json2edn:39963]        write:     114.21 ms
[json2edn:39963]      [total]:  20,309.50 ms

(note camel-snake-kebab missing from classpath, srctwo missing from classpath).

This is contrary to the behavior of clj in general, both the repl (:extra-paths/:extra-deps are how you introduce new code paths or deps) but also third party build tools like depstar:

jdnsmith ➤ clojure -A:srctwo:depstar -m hf.depstar.uberjar depstar.jar                                                                                                                                                                                                                                                                         git:master*
Building uber jar: depstar.jar
jdnsmith ➤ jar -tvf depstar.jar | (grep camel - > /dev/null && echo "csk in jar")                                                                                                                                                                                                                                                              git:master*
csk in jar
jdnsmith ➤ jar -tvf depstar.jar | (grep moresource - > /dev/null && echo "jdnsmith/moresource.clj in jar")                                                                                                                                                                                                                                     git:master*
jdnsmith/moresource.clj in jar

[lvh]

depstar appears to resolve this by looking at the current classpath: https://github.com/seancorfield/depstar/blob/master/src/hf/depstar/uberjar.clj#L208-L212

https://github.com/tonsky/uberdeps intentionally does not do that, but it's not clear to me why not doing that would be desirable.

[taylorwood]

Thanks for the detailed report!

depstar appears to resolve this by looking at the current classpath

This is also what I was doing originally, and I’ll likely revert to that.

https://github.com/taylorwood/clj.native-image/commit/0f113d46f9f0d07e8a29545c636431ddf5360a7d#diff-cacd55da9eaa0edb5d79e1e82917ed9dL7

[lvh]

Thanks for the prompt response! So, is this.. working as intended? Is there a different mechanism (as per uberdeps) to specify additional aliases?

It seems strange to me that a tool designed to build a classpath (clojure.tools.deps) makes so many other tools reconstitute details on the classpath :)

[lvh]

Oops: I misunderstood the change you linked; I see now that no, there's no obvious way to change the classpath via an alias and hence yeah, reverting to the old behavior seems appropriate to me -- but I'm new to the clojure.tools.deps ecosystem, maybe I'm just misunderstanding how this is supposed to work :)

[taylorwood]

I'm a bit reluctant to use the current tools.deps classpath for building the native image. It solves the issue for additional alias dependencies, but it also includes a lot of tooling stuff that's unnecessary to build the native image. For example with an example project, this is the classpath to build the native image:

classes:src:/Users/Taylor/.m2/repository/org/clojure/clojure/1.9.0/clojure-1.9.0.jar:/Users/Taylor/.m2/repository/org/clojure/data.json/0.2.6/data.json-0.2.6.jar:/Users/Taylor/.m2/repository/org/clojure/spec.alpha/0.1.143/spec.alpha-0.1.143.jar:/Users/Taylor/.m2/repository/org/clojure/core.specs.alpha/0.1.24/core.specs.alpha-0.1.24.jar

And it works fine (except for the additional alias issue).

If I use the current classpath it adds a ton of stuff that's unrelated/unnecessary to building the native image (but the additional alias works fine):

classes:src:/Users/Taylor/.m2/repository/javax/inject/javax.inject/1/javax.inject-1.jar:/Users/Taylor/.m2/repository/org/clojure/data.json/0.2.6/data.json-0.2.6.jar:/Users/Taylor/.m2/repository/org/clojure/clojure/1.9.0/clojure-1.9.0.jar:/Users/Taylor/.m2/repository/joda-time/joda-time/2.8.1/joda-time-2.8.1.jar:/Users/Taylor/.m2/repository/commons-codec/commons-codec/1.10/commons-codec-1.10.jar:/Users/Taylor/.m2/repository/org/codehaus/plexus/plexus-component-annotations/1.7.1/plexus-component-annotations-1.7.1.jar:/Users/Taylor/.m2/repository/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5.jar:/Users/Taylor/.m2/repository/org/springframework/build/aws-maven/5.0.0.RELEASE/aws-maven-5.0.0.RELEASE.jar:/Users/Taylor/.m2/repository/org/clojure/core.specs.alpha/0.1.24/core.specs.alpha-0.1.24.jar:/Users/Taylor/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.5.5/jackson-databind-2.5.5.jar:/Users/Taylor/.m2/repository/org/clojure/spec.alpha/0.1.143/spec.alpha-0.1.143.jar:/Users/Taylor/.m2/repository/org/clojure/tools.cli/0.3.5/tools.cli-0.3.5.jar:/Users/Taylor/.m2/repository/com/google/inject/guice/4.0/guice-4.0-no_aop.jar:/Users/Taylor/.m2/repository/org/apache/maven/resolver/maven-resolver-transport-wagon/1.1.1/maven-resolver-transport-wagon-1.1.1.jar:/Users/Taylor/.m2/repository/org/slf4j/jcl-over-slf4j/1.7.25/jcl-over-slf4j-1.7.25.jar:/Users/Taylor/.m2/repository/com/jcraft/jsch.agentproxy.jsch/0.0.9/jsch.agentproxy.jsch-0.0.9.jar:/Users/Taylor/.m2/repository/org/apache/maven/wagon/wagon-provider-api/3.0.0/wagon-provider-api-3.0.0.jar:/Users/Taylor/.m2/repository/com/jcraft/jsch.agentproxy.sshagent/0.0.9/jsch.agentproxy.sshagent-0.0.9.jar:/Users/Taylor/.m2/repository/com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.6.7/jackson-dataformat-cbor-2.6.7.jar:/Users/Taylor/.m2/repository/org/apache/maven/resolver/maven-resolver-transport-http/1.1.1/maven-resolver-transport-http-1.1.1.jar:/Users/Taylor/.m2/repository/org/apache/maven/maven-model-builder/3.5.2/maven-model-builder-3.5.2.jar:/Users/Taylor/.m2/repository/com/googlecode/javaewah/JavaEWAH/1.1.6/JavaEWAH-1.1.6.jar:/Users/Taylor/.m2/repository/org/codehaus/plexus/plexus-utils/3.1.0/plexus-utils-3.1.0.jar:/Users/Taylor/.m2/repository/org/apache/maven/resolver/maven-resolver-transport-file/1.1.1/maven-resolver-transport-file-1.1.1.jar:/Users/Taylor/.m2/repository/org/eclipse/sisu/org.eclipse.sisu.plexus/0.3.3/org.eclipse.sisu.plexus-0.3.3.jar:/Users/Taylor/.m2/repository/com/jcraft/jsch.agentproxy.usocket-jna/0.0.9/jsch.agentproxy.usocket-jna-0.0.9.jar:/Users/Taylor/.m2/repository/commons-io/commons-io/2.5/commons-io-2.5.jar:/Users/Taylor/.m2/repository/org/apache/maven/maven-settings-builder/3.5.2/maven-settings-builder-3.5.2.jar:/Users/Taylor/.m2/repository/org/clojure/tools.namespace/0.2.11/tools.namespace-0.2.11.jar:/Users/Taylor/.m2/repository/org/eclipse/jgit/org.eclipse.jgit/4.10.0.201712302008-r/org.eclipse.jgit-4.10.0.201712302008-r.jar:/Users/Taylor/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.5.5/jackson-core-2.5.5.jar:/Users/Taylor/.m2/repository/javax/enterprise/cdi-api/1.0/cdi-api-1.0.jar:/Users/Taylor/.m2/repository/org/apache/maven/maven-settings/3.5.2/maven-settings-3.5.2.jar:/Users/Taylor/.m2/repository/org/apache/httpcomponents/httpcore/4.4.8/httpcore-4.4.8.jar:/Users/Taylor/.m2/repository/org/apache/maven/maven-core/3.5.2/maven-core-3.5.2.jar:/Users/Taylor/.m2/repository/org/sonatype/plexus/plexus-cipher/1.4/plexus-cipher-1.4.jar:/Users/Taylor/.m2/repository/com/jcraft/jsch.agentproxy.pageant/0.0.9/jsch.agentproxy.pageant-0.0.9.jar:/Users/Taylor/.m2/repository/org/apache/maven/resolver/maven-resolver-api/1.1.1/maven-resolver-api-1.1.1.jar:/Users/Taylor/.m2/repository/software/amazon/ion/ion-java/1.0.2/ion-java-1.0.2.jar:/Users/Taylor/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.5.0/jackson-annotations-2.5.0.jar:/Users/Taylor/.m2/repository/javax/annotation/jsr250-api/1.0/jsr250-api-1.0.jar:/Users/Taylor/.m2/repository/org/apache/maven/maven-resolver-provider/3.5.2/maven-resolver-provider-3.5.2.jar:/Users/Taylor/.m2/repository/org/apache/maven/shared/maven-shared-utils/3.1.0/maven-shared-utils-3.1.0.jar:/Users/Taylor/.m2/repository/org/clojure/tools.deps.alpha/0.6.496/tools.deps.alpha-0.6.496.jar:/Users/Taylor/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:/Users/Taylor/.m2/repository/com/google/guava/guava/20.0/guava-20.0.jar:/Users/Taylor/.m2/repository/org/clojure/data.xml/0.2.0-alpha5/data.xml-0.2.0-alpha5.jar:/Users/Taylor/.m2/repository/com/amazonaws/jmespath-java/1.11.184/jmespath-java-1.11.184.jar:/Users/Taylor/.m2/repository/org/apache/maven/resolver/maven-resolver-spi/1.1.1/maven-resolver-spi-1.1.1.jar:/Users/Taylor/.m2/repository/com/amazonaws/aws-java-sdk-core/1.11.184/aws-java-sdk-core-1.11.184.jar:/Users/Taylor/.m2/repository/org/slf4j/slf4j-nop/1.6.2/slf4j-nop-1.6.2.jar:/Users/Taylor/.m2/repository/org/codehaus/plexus/plexus-classworlds/2.5.2/plexus-classworlds-2.5.2.jar:/Users/Taylor/.m2/repository/s3-wagon-private/s3-wagon-private/1.3.1/s3-wagon-private-1.3.1.jar:/Users/Taylor/.m2/repository/org/sonatype/plexus/plexus-sec-dispatcher/1.4/plexus-sec-dispatcher-1.4.jar:/Users/Taylor/.m2/repository/org/codehaus/plexus/plexus-interpolation/1.24/plexus-interpolation-1.24.jar:/Users/Taylor/.m2/repository/org/apache/httpcomponents/httpclient/4.5.4/httpclient-4.5.4.jar:/Users/Taylor/.m2/repository/net/java/dev/jna/jna/4.1.0/jna-4.1.0.jar:/Users/Taylor/.m2/repository/com/jcraft/jsch.agentproxy.connector-factory/0.0.9/jsch.agentproxy.connector-factory-0.0.9.jar:/Users/Taylor/.m2/repository/net/java/dev/jna/jna-platform/4.1.0/jna-platform-4.1.0.jar:/Users/Taylor/Projects/clj.native-image/src:/Users/Taylor/.m2/repository/org/clojure/tools.gitlibs/0.2.64/tools.gitlibs-0.2.64.jar:/Users/Taylor/.m2/repository/org/apache/maven/resolver/maven-resolver-connector-basic/1.1.1/maven-resolver-connector-basic-1.1.1.jar:/Users/Taylor/.m2/repository/org/apache/maven/resolver/maven-resolver-impl/1.1.1/maven-resolver-impl-1.1.1.jar:/Users/Taylor/.m2/repository/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.jar:/Users/Taylor/.m2/repository/org/apache/maven/maven-model/3.5.2/maven-model-3.5.2.jar:/Users/Taylor/.m2/repository/org/eclipse/sisu/org.eclipse.sisu.inject/0.3.3/org.eclipse.sisu.inject-0.3.3.jar:/Users/Taylor/.m2/repository/org/apache/maven/resolver/maven-resolver-util/1.1.1/maven-resolver-util-1.1.1.jar:/Users/Taylor/.m2/repository/camel-snake-kebab/camel-snake-kebab/0.4.0/camel-snake-kebab-0.4.0.jar:/Users/Taylor/.m2/repository/org/apache/maven/maven-repository-metadata/3.5.2/maven-repository-metadata-3.5.2.jar:/Users/Taylor/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar:/Users/Taylor/.m2/repository/org/apache/maven/maven-builder-support/3.5.2/maven-builder-support-3.5.2.jar:/Users/Taylor/.m2/repository/com/jcraft/jsch/0.1.54/jsch-0.1.54.jar:/Users/Taylor/.m2/repository/com/jcraft/jsch.agentproxy.core/0.0.9/jsch.agentproxy.core-0.0.9.jar:/Users/Taylor/.m2/repository/com/jcraft/jsch.agentproxy.usocket-nc/0.0.9/jsch.agentproxy.usocket-nc-0.0.9.jar:/Users/Taylor/.m2/repository/com/amazonaws/aws-java-sdk-kms/1.11.184/aws-java-sdk-kms-1.11.184.jar:/Users/Taylor/.m2/repository/org/apache/maven/maven-plugin-api/3.5.2/maven-plugin-api-3.5.2.jar:/Users/Taylor/.m2/repository/org/apache/maven/maven-artifact/3.5.2/maven-artifact-3.5.2.jar:/Users/Taylor/.m2/repository/org/clojure/data.codec/0.1.0/data.codec-0.1.0.jar:/Users/Taylor/.m2/repository/com/amazonaws/aws-java-sdk-s3/1.11.184/aws-java-sdk-s3-1.11.184.jar

I'm not sure if having all this "extra" stuff on the classpath would ever cause a problem or not when building the native image — it doesn't cause a problem with the example I tried so maybe my reluctance is unfounded. I'll do some experimentation and either revert my classpath building changes, or try to fix the alias issue specifically.

maybe I'm just misunderstanding how this is supposed to work

I'm not super confident in my understanding either! I do know that tools.deps itself is not meant to be a "build" tool but there are several projects (like this one) that are building "build" tools on top of it, so I guess we're all learning something 😄

I see this in the tools.deps docs, which makes me think I might be doing something wrong:

Compute the resolve-deps args If -R specifies one or more aliases, find each alias in the deps map :aliases merge-with merge the alias maps - the result is the resolve-args map

[lvh]

My understanding of the SubstratevM compiler is that it'll aggressively try to get rid of unnecessary classes, though that is predicated on them being AOT compiled in the first place so that they can be inspected :)

[taylorwood]

I figured it'd be kinda foolish to mimic what Clojure CLI is already doing with aliases, so I just reverted to using the clj-calculated classpath. The resulting images seem to be consistently, negligibly larger but 🤷‍♂