search

metabase / metabase

The simplest, fastest way to get business intelligence and analytics to everyone in your company :yum:
https://metabase.com
Other
38.41k stars 5.09k forks source link

Why is the new Spark SQL driver so B I G 🤯 #20543

Closed camsaul closed 2 years ago

camsaul commented 2 years ago

Before #20353 :

MB Uberjar was 265 MB (https://github.com/metabase/metabase/actions/runs/1848360089)

After #20353:

MB Uberjar is 346 MB (https://github.com/metabase/metabase/actions/runs/1848406617)

That's an 80 MB increase! 🙀

Let's see if we can tree shake the JDBC driver dep a bit.

camsaul commented 2 years ago

Here's everything that ends up getting included/skipped

SKIP    aopalliance/aopalliance (provided by metabase-core)
INCLUDE ch.qos.logback/logback-classic
INCLUDE ch.qos.logback/logback-core
INCLUDE co.cask.tephra/tephra-api
INCLUDE co.cask.tephra/tephra-core
INCLUDE co.cask.tephra/tephra-hbase-compat-1.0
INCLUDE com.carrotsearch/hppc
INCLUDE com.cedarsoftware/java-util
INCLUDE com.cedarsoftware/json-io
SKIP    com.fasterxml.jackson.core/jackson-annotations (provided by metabase-core)
SKIP    com.fasterxml.jackson.core/jackson-core (provided by metabase-core)
SKIP    com.fasterxml.jackson.core/jackson-databind (provided by metabase-core)
INCLUDE com.fasterxml.jackson.jaxrs/jackson-jaxrs-base
INCLUDE com.fasterxml.jackson.jaxrs/jackson-jaxrs-json-provider
INCLUDE com.fasterxml.jackson.module/jackson-module-jaxb-annotations
SKIP    com.fasterxml.woodstox/woodstox-core (provided by metabase-core)
INCLUDE com.github.joshelser/dropwizard-metrics-hadoop-metrics2-reporter
INCLUDE com.github.stephenc.findbugs/findbugs-annotations
INCLUDE com.github.stephenc.jcip/jcip-annotations
SKIP    com.google.code.findbugs/jsr305 (provided by metabase-core)
INCLUDE com.google.code.gson/gson
SKIP    com.google.guava/guava (provided by metabase-core)
SKIP    com.google.inject/guice (provided by metabase-core)
INCLUDE com.google.inject.extensions/guice-assistedinject
INCLUDE com.google.inject.extensions/guice-servlet
INCLUDE com.google.protobuf/protobuf-java
INCLUDE com.google.re2j/re2j
INCLUDE com.jcraft/jsch
INCLUDE com.jolbox/bonecp
INCLUDE com.lmax/disruptor
INCLUDE com.microsoft.sqlserver/mssql-jdbc
INCLUDE com.nimbusds/nimbus-jose-jwt
INCLUDE com.sun.jersey/jersey-client
INCLUDE com.sun.jersey/jersey-core
INCLUDE com.sun.jersey/jersey-json
INCLUDE com.sun.jersey/jersey-server
INCLUDE com.sun.jersey/jersey-servlet
INCLUDE com.sun.jersey.contribs/jersey-guice
INCLUDE com.sun.xml.bind/jaxb-impl
INCLUDE com.tdunning/json
INCLUDE com.thoughtworks.paranamer/paranamer
INCLUDE com.vlkan/flatbuffers
INCLUDE com.zaxxer/HikariCP
SKIP    com.zaxxer/HikariCP-java7 (provided by metabase-core)
INCLUDE commons-beanutils/commons-beanutils
INCLUDE commons-cli/commons-cli
SKIP    commons-codec/commons-codec (provided by metabase-core)
SKIP    commons-collections/commons-collections (provided by metabase-core)
INCLUDE commons-daemon/commons-daemon
INCLUDE commons-dbcp/commons-dbcp
SKIP    commons-io/commons-io (provided by metabase-core)
SKIP    commons-lang/commons-lang (provided by metabase-core)
SKIP    commons-net/commons-net (provided by metabase-core)
INCLUDE commons-pool/commons-pool
INCLUDE de.ruedigermoeller/fst
INCLUDE dnsjava/dnsjava
INCLUDE io.airlift/aircompressor
SKIP    io.dropwizard.metrics/metrics-core (provided by metabase-core)
INCLUDE io.dropwizard.metrics/metrics-json
INCLUDE io.dropwizard.metrics/metrics-jvm
INCLUDE io.netty/netty
INCLUDE io.netty/netty-all
INCLUDE io.netty/netty-buffer
INCLUDE io.netty/netty-codec
INCLUDE io.netty/netty-common
INCLUDE io.netty/netty-handler
INCLUDE io.netty/netty-resolver
INCLUDE io.netty/netty-transport
INCLUDE io.netty/netty-transport-native-epoll
INCLUDE io.netty/netty-transport-native-unix-common
INCLUDE it.unimi.dsi/fastutil
SKIP    jakarta.activation/jakarta.activation-api (provided by metabase-core)
INCLUDE javax.annotation/javax.annotation-api
INCLUDE javax.inject/javax.inject
INCLUDE javax.jdo/jdo-api
SKIP    javax.servlet/javax.servlet-api (provided by metabase-core)
INCLUDE javax.servlet.jsp/javax.servlet.jsp-api
INCLUDE javax.servlet.jsp/jsp-api
INCLUDE javax.transaction/jta
INCLUDE javax.transaction/transaction-api
INCLUDE javax.validation/validation-api
INCLUDE javax.ws.rs/javax.ws.rs-api
INCLUDE javax.ws.rs/jsr311-api
SKIP    javax.xml.bind/jaxb-api (provided by metabase-core)
INCLUDE javolution/javolution
INCLUDE jline/jline
SKIP    joda-time/joda-time (provided by metabase-core)
SKIP    junit/junit (provided by metabase-core)
INCLUDE log4j/log4j
INCLUDE net.minidev/accessors-smart
INCLUDE net.minidev/json-smart
INCLUDE net.sf.jpam/jpam
INCLUDE net.sf.opencsv/opencsv
INCLUDE org.antlr/antlr-runtime
INCLUDE org.apache.ant/ant
INCLUDE org.apache.ant/ant-launcher
INCLUDE org.apache.arrow/arrow-format
INCLUDE org.apache.arrow/arrow-memory
INCLUDE org.apache.arrow/arrow-vector
INCLUDE org.apache.avro/avro
SKIP    org.apache.commons/commons-collections4 (provided by metabase-core)
SKIP    org.apache.commons/commons-compress (provided by metabase-core)
INCLUDE org.apache.commons/commons-configuration2
INCLUDE org.apache.commons/commons-crypto
SKIP    org.apache.commons/commons-lang3 (provided by metabase-core)
SKIP    org.apache.commons/commons-math3 (provided by metabase-core)
INCLUDE org.apache.commons/commons-text
INCLUDE org.apache.curator/apache-curator
INCLUDE org.apache.curator/curator-client
INCLUDE org.apache.curator/curator-framework
INCLUDE org.apache.curator/curator-recipes
INCLUDE org.apache.derby/derby
INCLUDE org.apache.geronimo.specs/geronimo-jcache_1.0_spec
INCLUDE org.apache.hadoop/hadoop-annotations
INCLUDE org.apache.hadoop/hadoop-auth
INCLUDE org.apache.hadoop/hadoop-client
INCLUDE org.apache.hadoop/hadoop-common
INCLUDE org.apache.hadoop/hadoop-distcp
INCLUDE org.apache.hadoop/hadoop-hdfs
INCLUDE org.apache.hadoop/hadoop-mapreduce-client-app
INCLUDE org.apache.hadoop/hadoop-mapreduce-client-common
INCLUDE org.apache.hadoop/hadoop-mapreduce-client-core
INCLUDE org.apache.hadoop/hadoop-mapreduce-client-jobclient
INCLUDE org.apache.hadoop/hadoop-mapreduce-client-shuffle
INCLUDE org.apache.hadoop/hadoop-yarn-api
INCLUDE org.apache.hadoop/hadoop-yarn-client
INCLUDE org.apache.hadoop/hadoop-yarn-common
INCLUDE org.apache.hadoop/hadoop-yarn-registry
INCLUDE org.apache.hadoop/hadoop-yarn-server-applicationhistoryservice
INCLUDE org.apache.hadoop/hadoop-yarn-server-common
INCLUDE org.apache.hadoop/hadoop-yarn-server-resourcemanager
INCLUDE org.apache.hadoop/hadoop-yarn-server-web-proxy
INCLUDE org.apache.hadoop.thirdparty/hadoop-shaded-guava
INCLUDE org.apache.hadoop.thirdparty/hadoop-shaded-protobuf_3_7
INCLUDE org.apache.hbase/hbase-client
INCLUDE org.apache.hbase/hbase-common
INCLUDE org.apache.hbase/hbase-common$tests
INCLUDE org.apache.hbase/hbase-hadoop-compat
INCLUDE org.apache.hbase/hbase-hadoop2-compat
INCLUDE org.apache.hbase/hbase-http
INCLUDE org.apache.hbase/hbase-mapreduce
INCLUDE org.apache.hbase/hbase-metrics
INCLUDE org.apache.hbase/hbase-metrics-api
INCLUDE org.apache.hbase/hbase-prefix-tree
INCLUDE org.apache.hbase/hbase-procedure
INCLUDE org.apache.hbase/hbase-protocol
INCLUDE org.apache.hbase/hbase-protocol-shaded
INCLUDE org.apache.hbase/hbase-replication
INCLUDE org.apache.hbase/hbase-server
INCLUDE org.apache.hbase.thirdparty/hbase-shaded-miscellaneous
INCLUDE org.apache.hbase.thirdparty/hbase-shaded-netty
INCLUDE org.apache.hbase.thirdparty/hbase-shaded-protobuf
INCLUDE org.apache.hive/hive-classification
INCLUDE org.apache.hive/hive-common
INCLUDE org.apache.hive/hive-jdbc
INCLUDE org.apache.hive/hive-llap-client
INCLUDE org.apache.hive/hive-llap-common
INCLUDE org.apache.hive/hive-llap-common$tests
INCLUDE org.apache.hive/hive-llap-server
INCLUDE org.apache.hive/hive-llap-tez
INCLUDE org.apache.hive/hive-metastore
INCLUDE org.apache.hive/hive-serde
INCLUDE org.apache.hive/hive-service
INCLUDE org.apache.hive/hive-service-rpc
INCLUDE org.apache.hive/hive-shims
INCLUDE org.apache.hive/hive-standalone-metastore
INCLUDE org.apache.hive/hive-storage-api
INCLUDE org.apache.hive/hive-upgrade-acid
INCLUDE org.apache.hive.shims/hive-shims-0.23
INCLUDE org.apache.hive.shims/hive-shims-common
INCLUDE org.apache.hive.shims/hive-shims-scheduler
INCLUDE org.apache.htrace/htrace-core
INCLUDE org.apache.htrace/htrace-core4
SKIP    org.apache.httpcomponents/httpclient (provided by metabase-core)
SKIP    org.apache.httpcomponents/httpcore (provided by metabase-core)
INCLUDE org.apache.kerby/kerb-admin
INCLUDE org.apache.kerby/kerb-client
INCLUDE org.apache.kerby/kerb-common
INCLUDE org.apache.kerby/kerb-core
INCLUDE org.apache.kerby/kerb-crypto
INCLUDE org.apache.kerby/kerb-identity
INCLUDE org.apache.kerby/kerb-server
INCLUDE org.apache.kerby/kerb-simplekdc
INCLUDE org.apache.kerby/kerb-util
INCLUDE org.apache.kerby/kerby-asn1
INCLUDE org.apache.kerby/kerby-config
INCLUDE org.apache.kerby/kerby-pkix
INCLUDE org.apache.kerby/kerby-util
INCLUDE org.apache.kerby/kerby-xdr
INCLUDE org.apache.kerby/token-provider
SKIP    org.apache.logging.log4j/log4j-1.2-api (provided by metabase-core)
SKIP    org.apache.logging.log4j/log4j-api (provided by metabase-core)
SKIP    org.apache.logging.log4j/log4j-core (provided by metabase-core)
SKIP    org.apache.logging.log4j/log4j-slf4j-impl (provided by metabase-core)
INCLUDE org.apache.logging.log4j/log4j-web
INCLUDE org.apache.orc/orc-core
INCLUDE org.apache.orc/orc-shims
INCLUDE org.apache.parquet/parquet-hadoop-bundle
INCLUDE org.apache.taglibs/taglibs-standard-impl
INCLUDE org.apache.taglibs/taglibs-standard-spec
INCLUDE org.apache.thrift/libfb303
INCLUDE org.apache.thrift/libthrift
INCLUDE org.apache.twill/twill-api
INCLUDE org.apache.twill/twill-common
INCLUDE org.apache.twill/twill-core
INCLUDE org.apache.twill/twill-discovery-api
INCLUDE org.apache.twill/twill-discovery-core
INCLUDE org.apache.twill/twill-zookeeper
INCLUDE org.apache.yetus/audience-annotations
INCLUDE org.apache.zookeeper/zookeeper
INCLUDE org.apache.zookeeper/zookeeper-jute
SKIP    org.clojure/clojure (provided by metabase-core)
SKIP    org.clojure/core.specs.alpha (provided by metabase-core)
SKIP    org.clojure/spec.alpha (provided by metabase-core)
INCLUDE org.codehaus.jackson/jackson-core-asl
INCLUDE org.codehaus.jackson/jackson-jaxrs
INCLUDE org.codehaus.jackson/jackson-mapper-asl
INCLUDE org.codehaus.jackson/jackson-xc
INCLUDE org.codehaus.jettison/jettison
SKIP    org.codehaus.woodstox/stax2-api (provided by metabase-core)
INCLUDE org.datanucleus/datanucleus-api-jdo
INCLUDE org.datanucleus/datanucleus-core
INCLUDE org.datanucleus/datanucleus-rdbms
INCLUDE org.datanucleus/javax.jdo
INCLUDE org.eclipse.jdt.core.compiler/ecj
INCLUDE org.eclipse.jetty/apache-jsp
INCLUDE org.eclipse.jetty/apache-jstl
INCLUDE org.eclipse.jetty/jetty-annotations
INCLUDE org.eclipse.jetty/jetty-client
SKIP    org.eclipse.jetty/jetty-http (provided by metabase-core)
SKIP    org.eclipse.jetty/jetty-io (provided by metabase-core)
INCLUDE org.eclipse.jetty/jetty-jaas
INCLUDE org.eclipse.jetty/jetty-jndi
INCLUDE org.eclipse.jetty/jetty-plus
INCLUDE org.eclipse.jetty/jetty-rewrite
INCLUDE org.eclipse.jetty/jetty-runner
INCLUDE org.eclipse.jetty/jetty-security
SKIP    org.eclipse.jetty/jetty-server (provided by metabase-core)
INCLUDE org.eclipse.jetty/jetty-servlet
SKIP    org.eclipse.jetty/jetty-util (provided by metabase-core)
INCLUDE org.eclipse.jetty/jetty-util-ajax
INCLUDE org.eclipse.jetty/jetty-webapp
INCLUDE org.eclipse.jetty/jetty-xml
INCLUDE org.eclipse.jetty.toolchain/jetty-schemas
INCLUDE org.eclipse.jetty.websocket/websocket-api
INCLUDE org.eclipse.jetty.websocket/websocket-client
INCLUDE org.eclipse.jetty.websocket/websocket-common
INCLUDE org.eclipse.jetty.websocket/websocket-server
INCLUDE org.eclipse.jetty.websocket/websocket-servlet
INCLUDE org.ehcache/ehcache
INCLUDE org.fusesource.leveldbjni/leveldbjni-all
INCLUDE org.glassfish/javax.el
INCLUDE org.glassfish.hk2/hk2-api
INCLUDE org.glassfish.hk2/hk2-locator
INCLUDE org.glassfish.hk2/hk2-utils
INCLUDE org.glassfish.hk2/osgi-resource-locator
INCLUDE org.glassfish.hk2.external/aopalliance-repackaged
INCLUDE org.glassfish.hk2.external/javax.inject
INCLUDE org.glassfish.jersey.bundles.repackaged/jersey-guava
INCLUDE org.glassfish.jersey.containers/jersey-container-servlet-core
INCLUDE org.glassfish.jersey.core/jersey-client
INCLUDE org.glassfish.jersey.core/jersey-common
INCLUDE org.glassfish.jersey.core/jersey-server
INCLUDE org.glassfish.jersey.media/jersey-media-jaxb
INCLUDE org.glassfish.web/javax.servlet.jsp
SKIP    org.hamcrest/hamcrest-core (provided by metabase-core)
INCLUDE org.jamon/jamon-runtime
INCLUDE org.javassist/javassist
INCLUDE org.jruby.jcodings/jcodings
INCLUDE org.jruby.joni/joni
INCLUDE org.mortbay.jetty/jetty
INCLUDE org.mortbay.jetty/jetty-util
SKIP    org.ow2.asm/asm (provided by metabase-core)
INCLUDE org.ow2.asm/asm-commons
INCLUDE org.ow2.asm/asm-tree
SKIP    org.slf4j/slf4j-api (provided by metabase-core)
INCLUDE org.xerial.snappy/snappy-java
INCLUDE sqlline/sqlline
SKIP    xerces/xercesImpl (provided by metabase-core)
SKIP    xml-apis/xml-apis (provided by metabase-core)
INCLUDE xmlenc/xmlenc
camsaul commented 2 years ago

Fixed by #20563