search

benedekfazekas / mranderson

Dependency inlining and shadowing
Eclipse Public License 1.0
151 stars 14 forks source link

Inlining when inlined lib shares same root namespace as project can be problematic #88

Open lread opened 1 year ago

lread commented 1 year ago

Background

I've been trying to help Mor Gazith work out some issues he is having with MrAnderson on Slack.

Use case

We whittled down a problematic use case when attempting to inline full.json from full.aws.

The root namespaces of these projects match their library names. They both start with full.

Symptom

After inlining deps, we would expect target/srcdeps/full/aws to exist, but it does not.

Steps to reproduce

Clone full.aws

git clone https://github.com/fullcontact/full.aws.git
cd full.aws

Configure project.clj to inline full.json like so:

(defproject fullcontact/full.aws "1.0.0-SNAPSHOT"
  :description "Async Amazon Webservices client."
  :url "https://github.com/fullcontact/full.aws"
  :license {:name "Eclipse Public License - v 1.0"
            :url "http://www.eclipse.org/legal/epl-v10.html"
            :distribution :repo}
  :deploy-repositories [["releases" {:url "https://clojars.org/repo/" :creds :gpg}]]
  :dependencies [[org.clojure/clojure "1.10.3"]
                 [com.amazonaws/aws-java-sdk "1.12.270"]
                 [com.taoensso/faraday "1.11.4" ; DynamoDB sugar
                  :exclusions [com.amazonaws/aws-java-sdk-dynamodb joda-time]]
                 [fullcontact/full.http "1.1.0"]
                 ^:inline-dep [fullcontact/full.json "0.12.0"
                  :exclusions [com.fasterxml.jackson.core/jackson-core]]
                 [fullcontact/full.async "1.1.0"]
                 [fullcontact/full.core "1.1.1"
                  :exclusions [org.clojure/clojurescript]]
                 [javax.xml.bind/jaxb-api "2.4.0-b180830.0359"]]
  :release-tasks [["vcs" "assert-committed"]
                  ["change" "version" "leiningen.release/bump-version" "release"]
                  ["vcs" "commit"]
                  ["vcs" "tag" "--no-sign"]
                  ["deploy"]
                  ["change" "version" "leiningen.release/bump-version"]
                  ["vcs" "commit"]
                  ["vcs" "push"]]
  :plugins [[lein-midje "3.1.3"][thomasa/mranderson "0.5.4-SNAPSHOT"]]
  :profiles {:dev {:dependencies [[midje "1.7.0"]]}})

(added :plugins, and ^:inline-dep, also removed :aot :all to not muddy the waters with any effects it might have).

Inline the deps:

$ lein inline-deps             
project prefix:  mranderson43fbf21b
retrieve dependencies and munge clojure source files
in RESOLVED-TREE mode, working on a resolved dependency tree
 [fullcontact/full.json "0.12.0" :exclusions [[com.fasterxml.jackson.core/jackson-core]]]
   [camel-snake-kebab "0.4.0"]
   [cheshire "5.5.0"]
     [com.fasterxml.jackson.dataformat/jackson-dataformat-cbor "2.5.3"]
     [com.fasterxml.jackson.dataformat/jackson-dataformat-smile "2.5.3"]
     [tigris "0.1.1"]
   [fullcontact/camelsnake "0.10.0"]
   [fullcontact/full.core "1.1.0"]
     [me.moocar/logback-gelf "0.3" :exclusions [[org.slf4j/slf4j-api]]]
       [me.moocar/socket-encoder-appender "0.1beta1"]
     [org.slf4j/slf4j-api "1.7.21"]
     [org.clojure/tools.cli "0.3.5"]
     [ns-tracker "0.3.1"]
       [org.clojure/java.classpath "0.2.3"]
       [org.clojure/tools.namespace "0.2.11"]
     [org.yaml/snakeyaml "1.17"]
     [clj-yaml "0.4.0" :exclusions [[org.yaml/snakeyaml]]]
     [org.clojure/tools.logging "0.3.1"]
     [clj-time "0.15.0"]
       [joda-time "2.10"]
     [org.slf4j/jul-to-slf4j "1.7.21"]
     [ch.qos.logback/logback-classic "1.1.7" :exclusions [[org.slf4j/slf4j-api]]]
       [ch.qos.logback/logback-core "1.1.7"]
     [commons-codec "1.10"]
unzipping [ fulljson  [ v0v12v0 ]]
unzipping [ fullcore  [ v1v1v0 ]]
unzipping [ logback-gelf  [ v0v3 ]]
unzipping [ cheshire  [ v5v5v0 ]]
unzipping [ jackson-dataformat-smile  [ v2v5v3 ]]
unzipping [ logback-classic  [ v1v1v7 ]]
unzipping [ jul-to-slfj  [ v1v7v21 ]]
unzipping [ socket-encoder-appender  [ v0v1beta1 ]]
unzipping [ slfj-api  [ v1v7v21 ]]
unzipping [ tigris  [ v0v1v1 ]]
unzipping [ clj-time  [ v0v15v0 ]]
unzipping [ joda-time  [ v2v10 ]]
unzipping [ commons-codec  [ v1v10 ]]
unzipping [ ns-tracker  [ v0v3v1 ]]
unzipping [ javaclasspath  [ v0v2v3 ]]
unzipping [ clj-yaml  [ v0v4v0 ]]
unzipping [ snakeyaml  [ v1v17 ]]
unzipping [ jackson-dataformat-cbor  [ v2v5v3 ]]
unzipping [ toolsnamespace  [ v0v2v11 ]]
unzipping [ logback-core  [ v1v1v7 ]]
unzipping [ camelsnake  [ v0v10v0 ]]
unzipping [ camel-snake-kebab  [ v0v4v0 ]]
unzipping [ toolscli  [ v0v3v5 ]]
unzipping [ toolslogging  [ v0v3v1 ]]
  munge source files of toolslogging artifact on branch [] exposed false.
  munge source files of toolscli artifact on branch [] exposed false.
  munge source files of camel-snake-kebab artifact on branch [] exposed false.
  munge source files of camelsnake artifact on branch [] exposed false.
    prefixing imports in clojure files in 'target/srcdeps/camelsnake' ...
    prefixing imports: done
  munge source files of logback-core artifact on branch [] exposed false.
  munge source files of toolsnamespace artifact on branch [] exposed false.
  munge source files of jackson-dataformat-cbor artifact on branch [] exposed false.
  munge source files of snakeyaml artifact on branch [] exposed false.
  munge source files of clj-yaml artifact on branch [] exposed false.
    prefixing imports in clojure files in 'target/srcdeps/clj_yaml' ...
    prefixing imports: done
  munge source files of javaclasspath artifact on branch [] exposed false.
  munge source files of ns-tracker artifact on branch [] exposed false.
  munge source files of commons-codec artifact on branch [] exposed false.
  munge source files of joda-time artifact on branch [] exposed false.
  munge source files of clj-time artifact on branch [] exposed false.
    prefixing imports in clojure files in 'target/srcdeps/clj_time' ...
    prefixing imports: done
  munge source files of tigris artifact on branch [] exposed false.
    prefixing imports in clojure files in 'target/srcdeps/tigris' ...
    prefixing imports: done
  munge source files of slfj-api artifact on branch [] exposed false.
  munge source files of socket-encoder-appender artifact on branch [] exposed false.
  munge source files of jul-to-slfj artifact on branch [] exposed false.
  munge source files of logback-classic artifact on branch [] exposed false.
  munge source files of jackson-dataformat-smile artifact on branch [] exposed false.
  munge source files of cheshire artifact on branch [] exposed false.
    prefixing imports in clojure files in 'target/srcdeps/cheshire' ...
    prefixing imports: done
  munge source files of logback-gelf artifact on branch [] exposed false.
  munge source files of fullcore artifact on branch [] exposed false.
    prefixing imports in clojure files in 'target/srcdeps/full/core' ...
    prefixing imports: done
  munge source files of fulljson artifact on branch [] exposed false.
    prefixing imports in clojure files in 'target/srcdeps/full' ...
    prefixing imports: done
jaring all class file dependencies into target/class-deps.jar
prefixing #{"org" "ns_tracker" "clojure" "ch" "clj_time" "camelsnake" "cheshire" "clj_yaml" "full" "me" "com"} in target/class-deps.jar with fullaws100SNAPSHOT
deleting directories with class files in target/srcdeps...
   org  deleted
   ns_tracker  deleted
   clojure  deleted
   ch  deleted
   clj_time  deleted
   camelsnake  deleted
   cheshire  deleted
   clj_yaml  deleted
   full  deleted
   me  deleted
   com  deleted
unzipping repackaged class-deps.jar into target/srcdeps

Notice near the bottom of the output:

   full  deleted

And if we look at generated code you'll notice that yeah full it is not there:

$ ls -l target/srcdeps 
total 16
drwxrwxr-x 13 lee lee 4096 Jan  4 12:51 fullaws100SNAPSHOT
-rw-rw-r--  1 lee lee  992 Jan  4 12:51 JSONStringEscapingInputStream.class
drwxrwxr-x 16 lee lee 4096 Jan  4 12:51 mranderson43fbf21b
-rw-rw-r--  1 lee lee 1112 Jan  4 12:51 project.clj

Next Steps

I can take a look at a fix.

lread commented 1 year ago

Observation: the full libs seem to include.class files. Which is partly what triggers this behaviour, I suppose.

For example if I take a peek at the full.json jar:

$ unzip -l ~/.m2/repository/fullcontact/full.json/0.12.0/full.json-0.12.0.jar 
Archive:  /home/lee/.m2/repository/fullcontact/full.json/0.12.0/full.json-0.12.0.jar
  Length      Date    Time    Name
---------  ---------- -----   ----
      240  2020-02-03 09:44   META-INF/MANIFEST.MF
     3050  2020-02-03 09:44   META-INF/maven/fullcontact/full.json/pom.xml
     1606  2020-02-03 09:44   META-INF/leiningen/fullcontact/full.json/project.clj
     1077  2020-02-03 09:44   META-INF/leiningen/fullcontact/full.json/README.md
    11514  2020-02-03 09:44   META-INF/leiningen/fullcontact/full.json/LICENSE
        0  2020-02-03 09:44   full/
     2326  2020-02-03 09:44   full/json$write_json.class
     1181  2020-02-03 09:44   full/json$fn__920.class
      927  2020-02-03 09:44   full/json$get_preserved_keys$iter__888__892.class
     2701  2020-02-03 09:44   full/json$read_json.class
     1320  2020-02-03 09:44   full/json$fn__886.class
     1188  2020-02-03 09:44   full/json$fn__918.class
     1545  2020-02-03 09:44   full/json$convert_keys.class
     1190  2020-02-03 09:44   full/json$fn__916.class
      997  2020-02-03 09:44   full/json$slurp_json.class
     1001  2020-02-03 09:44   full/json$convert_map_keys$walk_fn__904.class
     1078  2020-02-03 09:44   full/json$convert_keys$fn__913.class
     1374  2020-02-03 09:44   full/json$get_preserved_keys.class
     3054  2020-02-03 09:44   full/json$get_preserved_keys$iter__888__892$fn__893.class
     3007  2020-02-03 09:44   full/json$loading__6721__auto____171.class
     1785  2020-02-03 09:44   full/json$convert_map_keys.class
     1013  2020-02-03 09:44   full/json$slurp_json_resource.class
     1867  2020-02-03 09:44   full/json$get_preserved_keys$iter__888__892$fn__893$fn__894.class
     5543  2020-02-03 09:44   full/json__init.class
     1385  2020-02-03 09:44   full/json$convert_map_keys$fn__909.class
        0  2020-02-03 09:44   clojure/
        0  2020-02-03 09:44   clojure/tools/
     1593  2020-02-03 09:44   clojure/tools/logging$eval869.class
     3332  2020-02-03 09:44   clojure/tools/logging$eval869$fn__874.class
     2670  2020-02-03 09:44   clojure/tools/logging$eval869$fn__870.class
     1529  2020-02-03 09:44   clojure/tools/logging$eval882$reify__883.class
      609  2020-02-03 09:44   clojure/tools/logging$eval882.class
        0  2020-02-03 09:44   META-INF/
        0  2020-02-03 09:44   META-INF/maven/
        0  2020-02-03 09:44   META-INF/maven/fullcontact/
        0  2020-02-03 09:44   META-INF/maven/fullcontact/full.json/
      147  2020-02-03 09:44   META-INF/maven/fullcontact/full.json/pom.properties
        0  2020-02-03 09:44   cheshire/
     5189  2020-02-03 09:44   cheshire/generate_seq$generate_key_fn_map.class
      600  2020-02-03 09:44   cheshire/generate_seq$eval563.class
      806  2020-02-03 09:44   cheshire/generate$encode_bool.class
    24391  2020-02-03 09:44   cheshire/generate$generate.class
     1581  2020-02-03 09:44   cheshire/generate_seq$write_start_object.class
     3329  2020-02-03 09:44   cheshire/core$with_writer.class
     1639  2020-02-03 09:44   cheshire/generate$encode_named.class
     2141  2020-02-03 09:44   cheshire/factory$loading__6721__auto____175.class
     5922  2020-02-03 09:44   cheshire/generate_seq$eval541$fn__542.class
      600  2020-02-03 09:44   cheshire/generate_seq$eval618.class
     2464  2020-02-03 09:44   cheshire/generate$eval276$fn__277.class
     1593  2020-02-03 09:44   cheshire/generate$i_QMARK_.class
      599  2020-02-03 09:44   cheshire/generate_seq$eval524.class
     2350  2020-02-03 09:44   cheshire/generate_seq$eval510$fn__511.class
     4884  2020-02-03 09:44   cheshire/factory$make_cbor_factory.class
     5343  2020-02-03 09:44   cheshire/generate$generate_map.class
      600  2020-02-03 09:44   cheshire/generate_seq$eval541.class
     4207  2020-02-03 09:44   cheshire/generate_seq$generate_array__631.class
      788  2020-02-03 09:44   cheshire/generate$write_string.class
      575  2020-02-03 09:44   cheshire/generate$eval276.class
      576  2020-02-03 09:44   cheshire/generate$eval233.class
     2668  2020-02-03 09:44   cheshire/generate_seq$generate_map__615.class
     1135  2020-02-03 09:44   cheshire/generate$encode_str.class
     5489  2020-02-03 09:44   cheshire/generate$eval255$fn__256.class
     2322  2020-02-03 09:44   cheshire/generate_seq$write_end_object__522.class
      730  2020-02-03 09:44   cheshire/generate$fn__207$G__203__210.class
     2171  2020-02-03 09:44   cheshire/generate$tag.class
     5134  2020-02-03 09:44   cheshire/generate$generate_basic_map__250.class
      599  2020-02-03 09:44   cheshire/generate_seq$eval584.class
     2299  2020-02-03 09:44   cheshire/core$parse_string_strict.class
     1888  2020-02-03 09:44   cheshire/generate$remove_encoder$fn__503.class
     5404  2020-02-03 09:44   cheshire/parse$eval796$fn__797.class
     2205  2020-02-03 09:44   cheshire/factory$fn__179.class
      558  2020-02-03 09:44   cheshire/parse$eval796.class
      576  2020-02-03 09:44   cheshire/generate$eval255.class
     2410  2020-02-03 09:44   cheshire/generate_seq$write_start_array__529.class
     2098  2020-02-03 09:44   cheshire/parse$parse_strict.class
     2713  2020-02-03 09:44   cheshire/generate$generate_array.class
     1549  2020-02-03 09:44   cheshire/generate_seq$write_end_object.class
     2552  2020-02-03 09:44   cheshire/parse$parse_object.class
    34215  2020-02-03 09:44   cheshire/generate_seq$generate.class
     4829  2020-02-03 09:44   cheshire/factory$make_json_factory.class
     4151  2020-02-03 09:44   cheshire/generate$encode_number.class
     2185  2020-02-03 09:44   cheshire/core$parse_smile.class
     2183  2020-02-03 09:44   cheshire/core$parse_stream.class
     1547  2020-02-03 09:44   cheshire/generate_seq$write_end_array.class
     1334  2020-02-03 09:44   cheshire/parse$fn__775.class
     1754  2020-02-03 09:44   cheshire/core$parsed_seq_STAR_$fn__862.class
     4110  2020-02-03 09:44   cheshire/core$generate_string.class
      576  2020-02-03 09:44   cheshire/generate$eval310.class
     1717  2020-02-03 09:44   cheshire/core$fn__828.class
     1968  2020-02-03 09:44   cheshire/generate$eval220$fn__221.class
     4887  2020-02-03 09:44   cheshire/factory$make_smile_factory.class
     5066  2020-02-03 09:44   cheshire/generate$eval233$fn__234.class
     5791  2020-02-03 09:44   cheshire/parse$parse_object__792.class
      599  2020-02-03 09:44   cheshire/generate_seq$eval510.class
     2262  2020-02-03 09:44   cheshire/generate_seq$eval517$fn__518.class
     3370  2020-02-03 09:44   cheshire/core$generate_cbor.class
     5995  2020-02-03 09:44   cheshire/generate_seq$generate_basic_map__558.class
     3132  2020-02-03 09:44   cheshire/generate_seq$loading__6721__auto____506.class
      732  2020-02-03 09:44   cheshire/generate$encode_nil.class
     2689  2020-02-03 09:44   cheshire/generate$encode_seq.class
     1811  2020-02-03 09:44   cheshire/generate_seq$fn__508.class
     1316  2020-02-03 09:44   cheshire/core$create_generator.class
      817  2020-02-03 09:44   cheshire/generate$encode_int.class
     2076  2020-02-03 09:44   cheshire/core$parsed_smile_seq.class
     3433  2020-02-03 09:44   cheshire/generate$generate_array__323.class
     1764  2020-02-03 09:44   cheshire/generate$fn__199.class
     5563  2020-02-03 09:44   cheshire/generate$generate_key_fn_map__274.class
     1903  2020-02-03 09:44   cheshire/parse$loading__6721__auto____773.class
     1832  2020-02-03 09:44   cheshire/generate$fn__207$G__202__214.class
     1784  2020-02-03 09:44   cheshire/factory$fn__177.class
    14933  2020-02-03 09:44   cheshire/core__init.class
     1271  2020-02-03 09:44   cheshire/generate$remove_encoder.class
     2572  2020-02-03 09:44   cheshire/parse$parse_array.class
     1717  2020-02-03 09:44   cheshire/generate$fn__204.class
     3378  2020-02-03 09:44   cheshire/core$generate_smile.class
     6260  2020-02-03 09:44   cheshire/generate_seq$eval563$fn__564.class
     2177  2020-02-03 09:44   cheshire/core$parse_cbor.class
     3247  2020-02-03 09:44   cheshire/core$loading__6721__auto____173.class
     2420  2020-02-03 09:44   cheshire/parse$parse.class
     2024  2020-02-03 09:44   cheshire/generate$write_string__225.class
     2038  2020-02-03 09:44   cheshire/factory$fn__188.class
        0  2020-02-03 09:44   cheshire/generate/
      173  2020-02-03 09:44   cheshire/generate/JSONable.class
     5100  2020-02-03 09:44   cheshire/factory__init.class
     8172  2020-02-03 09:44   cheshire/generate_seq__init.class
     2494  2020-02-03 09:44   cheshire/generate$loading__6721__auto____197.class
     2062  2020-02-03 09:44   cheshire/core$parsed_seq.class
     4133  2020-02-03 09:44   cheshire/generate_seq$eval618$fn__619.class
     3743  2020-02-03 09:44   cheshire/generate$generate_basic_map.class
     2412  2020-02-03 09:44   cheshire/generate_seq$write_start_object__515.class
     2933  2020-02-03 09:44   cheshire/core$write.class
     2055  2020-02-03 09:44   cheshire/factory$fn__191.class
    16808  2020-02-03 09:44   cheshire/generate__init.class
     6339  2020-02-03 09:44   cheshire/generate_seq$generate_key_fn_map__582.class
     3239  2020-02-03 09:44   cheshire/generate$fail.class
     4029  2020-02-03 09:44   cheshire/generate$encode_map.class
     5146  2020-02-03 09:44   cheshire/generate_seq$generate_basic_map.class
     2261  2020-02-03 09:44   cheshire/generate_seq$eval531$fn__532.class
      987  2020-02-03 09:44   cheshire/parse$lazily_parse_array.class
     2530  2020-02-03 09:44   cheshire/generate$generate_map__307.class
     3364  2020-02-03 09:44   cheshire/generate$eval310$fn__311.class
     3645  2020-02-03 09:44   cheshire/generate$generate_key_fn_map.class
     5463  2020-02-03 09:44   cheshire/parse$parse_array__804.class
     2051  2020-02-03 09:44   cheshire/factory$fn__194.class
     5906  2020-02-03 09:44   cheshire/parse__init.class
     2597  2020-02-03 09:44   cheshire/generate_seq$eval584$fn__585.class
      818  2020-02-03 09:44   cheshire/generate$encode_long.class
     2320  2020-02-03 09:44   cheshire/generate_seq$write_end_array__536.class
      821  2020-02-03 09:44   cheshire/generate$encode_ratio.class
     5729  2020-02-03 09:44   cheshire/parse$eval782$fn__783.class
     1451  2020-02-03 09:44   cheshire/generate$encode_date.class
     1443  2020-02-03 09:44   cheshire/generate$add_encoder.class
     2151  2020-02-03 09:44   cheshire/parse$tag.class
     2511  2020-02-03 09:44   cheshire/parse$lazily_parse_array$fn__806.class
      575  2020-02-03 09:44   cheshire/generate$eval220.class
     7395  2020-02-03 09:44   cheshire/generate_seq$generate_map.class
      917  2020-02-03 09:44   cheshire/core$parsed_seq_STAR_.class
      940  2020-02-03 09:44   cheshire/generate$fn__207.class
     4117  2020-02-03 09:44   cheshire/core$generate_stream.class
      599  2020-02-03 09:44   cheshire/generate_seq$eval531.class
      558  2020-02-03 09:44   cheshire/parse$eval782.class
     2278  2020-02-03 09:44   cheshire/core$parse_string.class
     1138  2020-02-03 09:44   cheshire/generate$encode_symbol.class
     8951  2020-02-03 09:44   cheshire/generate$number_dispatch.class
     2349  2020-02-03 09:44   cheshire/generate_seq$eval524$fn__525.class
     4774  2020-02-03 09:44   cheshire/parse$parse_STAR_.class
     4231  2020-02-03 09:44   cheshire/generate_seq$generate_array.class
      599  2020-02-03 09:44   cheshire/generate_seq$eval517.class
     1579  2020-02-03 09:44   cheshire/generate_seq$write_start_array.class
     3664  2020-01-31 10:18   full/json.cljc
---------                     -------
   495504                     170 files