pishen / sbt-lighter

SBT plugin for Apache Spark on AWS EMR
Apache License 2.0
57 stars 15 forks source link

Trying Mill & not sure why I am not getting #35

Open shy00n opened 3 years ago

shy00n commented 3 years ago

Hi:

Trying to use your wonderful tool for mill

  import mill.modules.Assembly
  import coursier.maven.MavenRepository

  //object cor_poc extends ScalaModule {
  object ai_io extends ScalaModule {
+   def name = "ai_io_grab_data"
    def scalaVersion = "2.12.10"
+   def sparkInstanceCount = 5
    def sparkAwsRegion = "us-east-1"
    def sparkS3JarFolder = "s3://hn-sandbox/emr_jarlys/"

    def repositories = super.repositories ++ Seq(
      MavenRepository("http://dl.bintray.com/spark-packages/maven"),
      MavenRepository("https://mvnrepository.com/artifact/org.apache.spark/spark-yarn"),
      MavenRepository("https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws"),
      MavenRepository("https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk")
    ),
     MavenRepository("https://mvnrepository.com/artifact")
     // net.pishen/sbt-lighter")
   )

(snip)...

    def ivyDeps = Agg(
      ivy"org.apache.spark::spark-core:3.0.1",
      ivy"org.apache.spark::spark-sql:3.0.1",
      ivy"org.apache.spark::spark-yarn:3.0.1",
      ivy"io.delta::delta-core:0.7.0",
      ivy"org.apache.hadoop:hadoop-aws:2.7.7",
     ivy"com.amazonaws:aws-java-sdk:1.7.4",
     ivy"net.pishen:sbt-lighter:1.2.0")

I suspect MavenRepository("https://mvnrepository.com/artifact") or MavenRepository("https://mvnrepository.com/artifact/net.pishen/sbt-lighter") is not right places to go. Can you tell me where I need point to? BTW: Also tried ivy"net.pishen::sbt-lighter:1.2.0" Also saw that publishMavenStyle := false in your sbt file. Darn~

Kind regards,

1 targets failed
ai_io.resolvedIvyDeps
Resolution failed for 1 modules:
--------------------------------------------
  net.pishen:sbt-lighter:1.2.0
        not found: /home/syoon/.ivy2/local/net.pishen/sbt-lighter/1.2.0/ivys/ivy.xml
        not found: https://repo1.maven.org/maven2/net/pishen/sbt-lighter/1.2.0/sbt-lighter-1.2.0.pom
        not found: http://dl.bintray.com/spark-packages/maven/net/pishen/sbt-lighter/1.2.0/sbt-lighter-1.2.0.pom
        not found: https://mvnrepository.com/artifact/org.apache.spark/spark-yarn/net/pishen/sbt-lighter/1.2.0/sbt-lighter-1.2.0.pom
        not found: https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/net/pishen/sbt-lighter/1.2.0/sbt-lighter-1.2.0.pom
        not found: https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk/net/pishen/sbt-lighter/1.2.0/sbt-lighter-1.2.0.pom
        not found: https://mvnrepository.com/artifact/net/pishen/sbt-lighter/1.2.0/sbt-lighter-1.2.0.pom
pishen commented 3 years ago

Haven't used mill before. But maybe you can try https://dl.bintray.com/sbt/sbt-plugin-releases/