search

azavea / osmesa

OSMesa is an OpenStreetMap processing stack based on GeoTrellis and Apache Spark
Apache License 2.0
79 stars 26 forks source link

Exclude AWS SDK JARs when creating an assembly #159

Open mojodna opened 4 years ago

mojodna commented 4 years ago

This shrinks the size of the assembly while also improving compatibility with EMR. When there are conflicts, Glue Catalog can't be used. When the AWS JARs are excluded, it works just fine (with provided versions of these libraries).

assemblyExcludedJars in assembly := {
  val cp = (fullClasspath in assembly).value
  cp filter { f =>
    f.data.getName.contains("aws-java-sdk")
  }
}

geotrellis-s3 (as gtS3) includes an exclude() in its dependencies; that may be able to go away using this approach.

(This should be applied to each of the projects here and probably ported to VP as well.)

CloudNiner commented 4 years ago

Possibly connected with #182 since we're migrating to AWS SDK v2 in that PR.