search

radanalyticsio / oshinko-s2i

This is a place to put s2i images and utilities for spark application builders for openshift
Apache License 2.0
15 stars 27 forks source link

Some plugins fail to download since they no longer support Scala 2.10 #293

Open toxicafunk opened 5 years ago

toxicafunk commented 5 years ago

On project/plugins.sbt:

addCompilerPlugin("com.olegpy" %% "better-monadic-for" % "0.3.0")

The above fails because better-monadic-for no longer as a "better-monadic-for_2.10" version.

This can be fixed specifying the version:

addCompilerPlugin("com.olegpy" % "better-monadic-for_2.11" % "0.3.0")

but it creates a problem if on you local machine you have scala 2.12:

java.lang.NoClassDefFoundError: scala/tools/nsc/transform/Transform$class

Maybe its time to update the scala and sbt versions of the base image since more plugins/libraries are dropping 2.10 support, i.e. sbt-wartremover and sbt-scalafmt,

elmiko commented 5 years ago

thanks for reporting this, i am going to ping a few of our scala experts.

@erikerlandson @willb any thoughts about this?

toxicafunk commented 5 years ago

Hi, any plans for an update? I could probably submit a PR if someone can point me in the right direction, i.e. which image is responsible for the build, etc.

Thanks

elmiko commented 5 years ago

hi @toxicafunk , i have talked about this very briefly with @erikerlandson and @willb. i would still love to get their comments here, but if you would like to take a stab at updating the scala version i'm happy to help.

the images that are generated from this project are created through a slightly convoluted process. if we are talking specifically about updating the scala build version and process for that type of build then i think we need to start by adding a new version here https://github.com/radanalyticsio/s2i-scala-container

that repo is the builder image that does the heavy lifting for scala. after it has been updated to a new scala version, then we might need to address the files in the modules/scala directory here. lastly, we will need to rebuild the image, which involves using cekit and the makefiles here to generate the dockerfiles, then we have autobuilds to read those.

toxicafunk commented 5 years ago

Hi @elmiko,

Thanks for your prompt response. It looks like radanalyticsio/s2i-scala-container is already prepared to run 2.11.8. So I will focus on module/scala to see why its not being used.

elmiko commented 5 years ago

it all starts here

https://github.com/radanalyticsio/oshinko-s2i/blob/master/image.scala.yaml#L6

but perhaps there is some issue where we are not building a proper image from the 2.11 stuff in that repo. i'm not sure

toxicafunk commented 4 years ago

The issue seems to be related with using sbt 0.13 which downloads scala 2.10 , upgrading to sbt 1.2x seems to fix the problem:

Using sbt 0.13 Getting Scala 2.10.6 (for sbt)...

Using 1.2.1 Getting Scala 2.11.6 (for sbt)...

Not sure the proper way to address this in a PR.

I'm currently working on a PR for using spark 2.4.4 with Scala 2.12 and Hadoop 2.8.5 though.

tmckayus commented 4 years ago

hmm, I may have just run into this in a PR build. I'll give it a try

toxicafunk commented 4 years ago

Here's the PR as promised: https://github.com/radanalyticsio/oshinko-s2i/pull/298