search

ondra-m / ruby-spark

Ruby wrapper for Apache Spark
MIT License
227 stars 29 forks source link

Install issues #21

Open librarywebchic opened 8 years ago

librarywebchic commented 8 years ago

I'm having issues with getting ruby spark installed on my Mac with Yosemite. When I run the command ruby-spark build

I get an error about Error: Invalid or corrupt jarfile sbt/sbt-launch-0.13.7.jar

I installed sbt with homebrew and it seems to be working but I always get this error.

Any suggestions on how to fix this and get ruby-spark properly running?

ondra-m commented 8 years ago

Please try clone repository and run bin/ruby-spark ....

librarywebchic commented 8 years ago

Ok so I ran

bin/ruby-spark build

Got Attempting to fetch sbt

Launching sbt from sbt/sbt-launch-0.13.9.jar Error: Invalid or corrupt jarfile sbt/sbt-launch-0.13.9.jar error: Spark cannot be assembled.. Use --trace to view backtrace

When I run it with the --trace option

Launching sbt from sbt/sbt-launch-0.13.9.jar Error: Invalid or corrupt jarfile sbt/sbt-launch-0.13.9.jar /Users/coombsk/git/my-ruby-spark/lib/spark/build.rb:37:in block in build': Spark cannot be assembled. (Spark::BuildError) from /Users/coombsk/git/my-ruby-spark/lib/spark/build.rb:35:inchdir' from /Users/coombsk/git/my-ruby-spark/lib/spark/build.rb:35:in build' from /Users/coombsk/git/my-ruby-spark/lib/spark/cli.rb:40:inblock (2 levels) in run' from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/command.rb:178:in call' from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/command.rb:178:incall' from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/command.rb:153:in run' from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/runner.rb:428:inrun_active_command' from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/runner.rb:68:in run!' from /Library/Ruby/Gems/2.0.0/gems/commander-4.3.5/lib/commander/delegates.rb:15:inrun!' from /Users/coombsk/git/my-ruby-spark/lib/spark/cli.rb:157:in run' from bin/ruby-spark:8:in

'

ondra-m commented 8 years ago

Go first to ext/spark/sbt and remove all sbt*.jar.

If that wont help:

  1. cd ext/spark
  2. rm sbt/sbt-*.jar
  3. sbt/sbt

Is sbt working?

librarywebchic commented 8 years ago

No I get an error when I run sbt/sbt

Launching sbt from sbt/sbt-launch-0.13.9.jar Error: Invalid or corrupt jarfile sbt/sbt-launch-0.13.9.jar

Karen

On Sun, Nov 15, 2015 at 1:18 AM, Ondřej Moravčík notifications@github.com wrote:

Go first to ext/spark/sbt and remove all sbt*.jar.

If that wont help:

  1. cd ext/spark
  2. rm sbt/sbt-*.jar
  3. sbt/sbt

Is sbt working?

— Reply to this email directly or view it on GitHub https://github.com/ondra-m/ruby-spark/issues/21#issuecomment-156788091.

jsmestad commented 8 years ago

I am having this same issue; following along, I get the same error via git clone or install from ruby gems.

Issue turned out to be how the sbt file is fetched. I downloaded the launcher file manually, renamed it, and everything works.

ondra-m commented 8 years ago

Please try

cd ext/spark

rm sbt/sbt-launch-0.13.9.jar

wget "http://dl.bintray.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.9/sbt-launch.jar" -O sbt/sbt-launch-0.13.9.jar

sbt/sbt

Do you see sbt console?

jsmestad commented 8 years ago

I also wanted to ask, if I already have spark, sbt, and scala installed via homebrew can ruby-spark just use that? - Maybe I misunderstand what exactly ruby-spark build does.

ondra-m commented 8 years ago

No. Everything is built on selected versions.

brightbytes-dude commented 8 years ago

I was experiencing this same issue with ruby-spark build on OSX Mavericks, and saw that the cause was that sbt/sbt-launch-0.13.9.jar was 0B in size.

Examining the sbt/sbt bash script provided the source $URL1 from which to download the jar, so I downloaded it manually, and it came down as 1.2MB (suggesting there's something wrong with the curl command for OSX).

I then moved the downloaded file to sbt/sbt-launch-0.13.9.jar and retried ruby-spark build, and the following occured:

Launching sbt from sbt/sbt-launch-0.13.9.jar
Getting org.scala-sbt sbt 0.13.9 ...

:: problems summary ::
:::: WARNINGS
        module not found: org.scala-sbt#sbt;0.13.9

    ==== local: tried

      /Users/aaron/.ivy2/local/org.scala-sbt/sbt/0.13.9/ivys/ivy.xml

      -- artifact org.scala-sbt#sbt;0.13.9!sbt.jar:

      /Users/aaron/.ivy2/local/org.scala-sbt/sbt/0.13.9/jars/sbt.jar

    ==== jcenter: tried

      https://jcenter.bintray.com/org/scala-sbt/sbt/0.13.9/sbt-0.13.9.pom

      -- artifact org.scala-sbt#sbt;0.13.9!sbt.jar:

      https://jcenter.bintray.com/org/scala-sbt/sbt/0.13.9/sbt-0.13.9.jar

    ==== typesafe-ivy-releases: tried

      https://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt/0.13.9/ivys/ivy.xml

    ==== Maven Central: tried

      https://repo1.maven.org/maven2/org/scala-sbt/sbt/0.13.9/sbt-0.13.9.pom

      -- artifact org.scala-sbt#sbt;0.13.9!sbt.jar:

      https://repo1.maven.org/maven2/org/scala-sbt/sbt/0.13.9/sbt-0.13.9.jar

... and some more useless debug stuff thereafter.

When I tried $URL2 from the script in the browser, I get a 404, so that source is no longer valid.

I am able to use brew to install sbt and apache-spark, but it's not clear how to use them having installed them thusly.

Any assistance would be appreciated!

ondra-m commented 8 years ago

Take a look on

https://github.com/sbt/sbt/issues/2295 http://stackoverflow.com/questions/6784463/error-trustanchors-parameter-must-be-non-empty/6788682