update to scala 2.10

[MasseGuillaume]

I'm in a quest to update to scala 2.10

paulp did a huge refactor on the repl ( https://github.com/scala/scala/commit/48cc8b47fcadaa187026ca0422178c9094e4b412#src/compiler/scala/tools/nsc )

this is the dependency graph I got from https://github.com/jrudolph/sbt-dependency-graph + dot and the updates I have to make on dependencies. Also I found some bugs on the dependencies like

• https://github.com/Netflix/RxJava/issues/192
• observable depends on rxjava-core 0.5.0 ( should be 0.5.3 )
• server depends on unfiltered-json that depends on lift-json but common depends directly on lift-json. If you put unfiltered and lift json versions out of synch you will have some transitive dependencies. ( so common should depends on unfiltered-json )

[KenCoder]

Great - 2.10 would be great, but we were worried about our dependency on the REPL.

I made the dependency updates

I'd like to get off RxJava - likely move to Scala-react once https://github.com/ingoem/scala-react/issues/4 is addressed.

[MasseGuillaume]

just got it to compile and it's running :+1:

got rid of HackIMain, I will fix it later.

I will push on a new branch https://github.com/MasseGuillaume/scala-notebook/tree/scala-2.10.0-update

[copumpkin]

Didn't 2.10 remove the ability to interrupt the repl?

[benjchristensen]

Curious, is there some aspect of RxJava that doesn't work well with Scala (https://github.com/Netflix/RxJava/tree/master/language-adaptors/rxjava-scala) that we should fix?

We haven't put any effort thus far into making it "idiomatic" so it relies upon java interop, so I imagine there is a lot that can be improved.

[KenCoder]

I'll try to redo some of the work and give you a better answer, but here are some things I recall. Maybe I missed the right way to do these things:

1. Lack of map operator - easy way to convert an Observer[A] to an Observer[B](or Observable)
2. Observable is only a concrete class, not an interface. So it was hard to define my own observables, then have code that could take them or others and behave in similar ways.
3. I don't recall exactly where, but I found the RxJava->Scala adapters didn't work with function type inference. So I would be doing things like (x:T) => foo(x)
4. Methods that were overloaded to take Object interfere with any sort of implicit type conversion. This might have been the same as 3.

[benjchristensen]

1) map operator

This is one of the most important and most commonly used operators in Rx.

Take a look at this wiki page: https://github.com/Netflix/RxJava/wiki/Transformative-Operators

You can also see the javadoc for the map operator (and many others) here: http://netflix.github.com/RxJava/javadoc/rx/Observable.html

map

public <R> Observable<R> map(Func1<T,R> func)
Applies a function of your choosing to every item emitted by an Observable, and returns this transformation as a new Observable sequence.

2) Observable being an abstract class rather than interface

Agreed this isn't ideal, but this is on purpose and a result of us first doing it as an interface internally and it not working because Java lacks extension methods. This was thought on and argued about for a long time after a year of using it internally before opensourcing it.

Every Observable needs the dozens of operators implemented so all of them ended up having to always extend of AbstractObservable anyways.

The correct way to easily create an Observable is using Observable.create().

You can see information on creating Observables here: https://github.com/Netflix/RxJava/wiki/Creation-Operators

Some examples:

• Groovy: https://github.com/Netflix/RxJava/blob/master/language-adaptors/rxjava-groovy/src/examples/groovy/rx/lang/groovy/examples/VideoExample.groovy#L109
• Java: https://github.com/benjchristensen/WSPerfLab/blob/master/ws-impls/ws-java-servlet-blocking-rxjava/src/main/java/perf/test/TestCaseAServlet.java#L160

Unfortunately I don't have a good example in Scala using Observable.create but you'll get the idea from those above.

This usage of Observable.create is following the Rx.Net model http://msdn.microsoft.com/en-us/library/hh229096(v=vs.103).aspx

3) Scala Type Inference

Can you please file a but with the example code (https://github.com/Netflix/RxJava/issues)? We just got a new member of our team who is more experience with Scala than I am and I can get him to help out if you can provide the bug report.

4) Object type

The object type is being used because it allows various different languages to all be supported - such as dynamic languages like JRuby, Clojure and Groovy.

You can learn more about how this works here: https://github.com/Netflix/RxJava/tree/master/language-adaptors

The use of Object is only ever used in places where a function is expected - a lambda/closure of some type that implements a Func/Action interface.

The desire is for a true polyglot system to be capable where Java/Scala/Clojure/JRuby/Groovy/etc can use rx.Observable and pass it around without having conversions from ScalaObservable to ClojureObservable etc.

The RxJava library has specifically been written to target the JVM and not a specific language so each language adaptor is supported right from the core library.

If you can suggest a better way of allowing various languages to transparently be supported without each of them needing to awkwardly conform to an arbitrary interface rather than just using their various closures/lambas I'd love to hear it.

[KenCoder]

Right, thanks. It was actually Observer.map that I was missing, as I'm trying to create an object that both publishes and consumes and can map. However, I can see the reasoning to providing everything on Observable. Let me try using it some more and I'll report back. I do think there are still some problems with the Object thing, but I'll let you know the specifics.

[benjchristensen]

No operators are ever done on the Observer itself, that is the callback. What you likely want is a Subject as it is both an Observable and Observer.

We've only got one Subject so far, but it may give you what you want: http://netflix.github.com/RxJava/javadoc/rx/subjects/Subject.html

You can read more about subjects here: http://msdn.microsoft.com/en-us/library/hh242970(v=vs.103).aspx

Github issue for the implementation of the rest: https://github.com/Netflix/RxJava/issues/17

[benjchristensen]

Here's the discussion on why Object is being used for language adaptors and whether there is a better way of doing it while not breaking the polyglot interop: https://groups.google.com/forum/#!topic/rxjava/bVZoKSsb1-o

[MasseGuillaume]

@benjchristensen I'm sorry about the "bad netflix eng." comment in the diagram, sometime with to much time in front of the computer I forgot that their is humans behind those projects :P.

@KenCoder I'm a bit stuck with migrating the test to scala 2.10 because of akka. I'm doing a small akka project so I can understand what is going on.

@copumpkin yep I saw a thread on google group about it. I will focus on getting the test working then I will see what I can do about this.

[benjchristensen]

@MasseGuillaume Not a problem. The comments in this thread have helped trigger improvements in RxJava so that's great.

The com.google.code.findbugs dependency is now removed as of pull request https://github.com/Netflix/RxJava/pull/211 as the usage of the JSR 305 annotations were trivial and not worth the dependency or confusion.

Better support of Scala via the use of implicits is very close to be submitted as a new colleague on my team has a lot of experience with Scala. He's providing updates at https://github.com/Netflix/RxJava/issues/208

It will be held up from being merged into master until https://github.com/Netflix/RxJava/issues/204 is solved which already has a prototype and just needs me or someone else to do the busy work of getting the Gradle build process hooked up.

[joshmarcus]

What's the status of this ticket? I am eager to use scala-notebook with our map analysis platform (see: github.com/geotrellis/geotrellis) which is currently on 2.10. Is there a pull request that would be helpful at this point?

[MasseGuillaume]

@joshmarcus my fork is compiling and working, I'm migrating the test part.

[copumpkin]

I'd assume it would help you if the tests actually worked :) or do they for you? It seems to vary based on where/how you run them, which is a tad suboptimal.

On Thu, Mar 28, 2013 at 7:04 PM, Guillaume Massé notifications@github.comwrote:

@joshmarcus https://github.com/joshmarcus my fork is compiling and working, I'm migrating the test part.

— Reply to this email directly or view it on GitHubhttps://github.com/Bridgewater/scala-notebook/issues/2#issuecomment-15620676 .

[MasseGuillaume]

@copumpkin well if he just want to try it's working :-).

I got the tests to compile https://github.com/MasseGuillaume/scala-notebook/commit/c2b65fa63efcac7e133e2c7a37cd89abbd355a26. It look like the migration broke the repl completion :-(.

@KenCoder @krisher any updates on https://github.com/Bridgewater/scala-notebook/issues/5 ?

[joshmarcus]

any new update on this?

[copumpkin]

Ack, I missed @MasseGuillaume's update on this. I'd love to update to 2.10 but does the completion breakage look fundamental? I expected to lose interruptibility but completion makes me sad. We should probably find a more general way of doing completion anyway, but for now it'd be nice to keep. I'll try tinkering with it a bit.

[MasseGuillaume]

I'm sorry, I had my final exams + I'm helping Andrew Phillips & Nermin Serifovic for their talk (Scala Puzzlers Reloaded at Scala Days). I'm rewriting ScalaKata as a service. I will focus when I'm done with my refactor.

[joshmarcus]

I can take a look at repl completion and see if I can submit a pull request that resolves the issue, if that would be helpful -- or take a look at #5 .

[copumpkin]

@MasseGuillaume no problem. I'm looking at your work but am seeing some local conflicts due to other commits updating versions and such. Will keep poking at that. Out of curiosity, why the rename from build.scala to NotebookBuild.scala?

@joshmarcus I think we might want to think of a more general approach to completion so it might not be a huge deal if we lose it temporarily. Also, @KenCoder's rework of the back-end might make the failing tests in #5 irrelevant.

[copumpkin]

I have a merge of @MasseGuillaume's 2.10 work with @KenCoder's latest back-end design sitting in https://github.com/copumpkin/scala-notebook. No pull request just yet because the tests are still hanging :( but otherwise it seems to work. Still needs some cleanup, too.

[joshmarcus]

Excellent! I am excited and I will test it out.

[copumpkin]

Thanks! I haven't had a chance to give it a decent run-through so it's quite possible I messed something up.

[joshmarcus]

Just checking in -- any progress on upgrading to 2.10?

[MasseGuillaume]

ping @KenCoder. I think they have a 2.10 internally already.

[KenCoder]

Yes, I'll work on it this weekend.

Ken

On Sep 5, 2013, at 10:34 AM, Guillaume Massé notifications@github.com wrote:

ping @KenCoder. I think they have a 2.10 internally already.

— Reply to this email directly or view it on GitHub.

[antonkulaga]

So, how soon are going to move to scala 2.10.x and sbt 0.13?

[KenCoder]

I've been pretty busy, but will try to spend some more time in the next week or so giving this project some loving again. I think copumpkin did most of the work, so mostly it needs integration and getting the build green again.

On Sun, Oct 20, 2013 at 8:41 PM, Anton Kulaga notifications@github.comwrote:

So, how soon are going to move to scala 2.10.x ?

— Reply to this email directly or view it on GitHubhttps://github.com/Bridgewater/scala-notebook/issues/2#issuecomment-26687297 .

[vincentk]

Lovely to see that this would seem to be going ahead. Perhaps there's something half-way working which you can already share?

[Bridgewater]

Just pushed the branch I'm working on to my repo at https://github.com/KenCoder/scala-notebook-1.git on the scala2.10 branch.

Wouldn't describe it as half-way - basically I've upgraded everything (akka, unfiltered, etc) to the latest version and have it compiling. I'm now working through the issues - mostly around the remote actor system stuff.

[KenCoder]

Status update - have most unit tests passing, but hitting a strange Akka issue. Works from IntelliJ, but when run from SBT the client actor gets a "Disassociated" message with no explanation and the calc doesn't go through. Banging my head against this now, irrationally thinking about changing to Finagle. Any Akka experts out there?

[MasseGuillaume]

@KenCoder same frustration here ! from akka https://github.com/CodeBrew-io/Sauce/commit/19f9ed7bfb729ce29a5d13afb39ded9acfe662c6 to finagle https://github.com/CodeBrew-io/Sauce/commit/5c0b8ec89cdd3cd8c95635edcb3f6cb89c8976f4 by the way check out the improved ScalaKata project: https://codebrew.io :-)

[KenCoder]

Thanks, this was super helpful, especially in locating the SBT 0.13 version of scrooge. I'm going to play with finagle a bit.

On Wed, Nov 27, 2013 at 6:08 PM, Guillaume Massé notifications@github.comwrote:

@KenCoder https://github.com/KenCoder same frustration here ! from akka CodeBrew-io/Sauce@19f9ed7https://github.com/CodeBrew-io/Sauce/commit/19f9ed7bfb729ce29a5d13afb39ded9acfe662c6 to finagle CodeBrew-io/Sauce@5c0b8echttps://github.com/CodeBrew-io/Sauce/commit/5c0b8ec89cdd3cd8c95635edcb3f6cb89c8976f4

by the way check out the improved ScalaKata project: https://codebrew.io

— Reply to this email directly or view it on GitHubhttps://github.com/Bridgewater/scala-notebook/issues/2#issuecomment-29427840 .

[MasseGuillaume]

@KenCoder try g8 MasseGuillaume/scrooge http://stackoverflow.com/questions/17332061/how-do-i-install-and-use-scrooge/17333542#17333542

[joshmarcus]

I know akka -- I could look at the akka issue if it's still pressing.

On Tuesday, December 3, 2013, Guillaume Massé wrote:

@KenCoder https://github.com/KenCoder try g8 MasseGuillaume/scrooge

— Reply to this email directly or view it on GitHubhttps://github.com/Bridgewater/scala-notebook/issues/2#issuecomment-29684269 .

[KenCoder]

Josh - it would be great if you could look at it.

Clone https://github.com/KenCoder/scala-notebook-1 Build in SBT 0.13. run project server Start a new notebook, and type 1+1 as an expression sn-server.log and sn-session.log show logs Running from SBT, it doesn't work and I get "Disassociated [akka.tcp:// Remote@127.0.0.1:56190] <- [akka.tcp://NotebookServer@127.0.0.1:56032]" in the logs Running from intelliJ (using gen-idea) it works.

Thanks!

On Tue, Dec 3, 2013 at 9:07 AM, Josh Marcus notifications@github.comwrote:

I know akka -- I could look at the akka issue if it's still pressing.

On Tuesday, December 3, 2013, Guillaume Massé wrote:

@KenCoder https://github.com/KenCoder try g8 MasseGuillaume/scrooge

— Reply to this email directly or view it on GitHub< https://github.com/Bridgewater/scala-notebook/issues/2#issuecomment-29684269>

.

— Reply to this email directly or view it on GitHubhttps://github.com/Bridgewater/scala-notebook/issues/2#issuecomment-29711696 .

[windemut]

Ken: I was trying to do this, but it is not working for me. It fails trying to download the scrooge-sbt-plugin, for which there apparently is no port to SBT 0.13 yet. Sbt 0.12 won't run for me because of some JLine incompatibility thing, and your instructions say to use SBT 0.13.

What am I missing?

[KenCoder]

I had pushed some Thrift stuff since i sent the prior email. Can you try the akka-2.10 branch, which I branched before the thrift changes. Thanks again

On Sat, Dec 21, 2013 at 12:15 PM, windemut notifications@github.com wrote:

Ken: I was trying to do this, but it is not working for me. It fails trying to download the scrooge-sbt-plugin, for which there apparently is no port to SBT 0.13 yet. Sbt 0.12 won't run for me because of some JLine incompatibility thing, and your instructions say to use SBT 0.13.

What am I missing?

— Reply to this email directly or view it on GitHubhttps://github.com/Bridgewater/scala-notebook/issues/2#issuecomment-31067385 .

[windemut]

There does not appear to be an akka-2.10 branch. I tried the scala2.10 branch again, and this time it started out with SBT set to scala-version = 2.10 (previously it was still set to 2.9.2). But, the issue with scrooge still appears to be there...

london:external andreas$ git clone -b scala2.10 https://github.com/KenCoder/scala-notebook-1 Cloning into 'scala-notebook-1'... remote: Counting objects: 1384, done. remote: Compressing objects: 100% (759/759), done. remote: Total 1384 (delta 380), reused 1307 (delta 317) Receiving objects: 100% (1384/1384), 702.84 KiB | 51.00 KiB/s, done. Resolving deltas: 100% (380/380), done. Checking connectivity... done london:external andreas$ cd scala-notebook-1/ london:scala-notebook-1 andreas$ sbt [info] Loading project definition from /Users/andreas/code/external/scala-notebook-1/project [warn] Multiple resolvers having different access mechanism configured with same name 'sbt-plugin-releases'. To avoid conflict, Remove duplicate project resolvers (resolvers) or rename publishing resolver (publishTo). [info] Updating {file:/Users/andreas/code/external/scala-notebook-1/project/}scala-notebook-1-build... [info] Resolving com.twitter#scrooge-sbt-plugin;3.9.0 ... [warn] module not found: com.twitter#scrooge-sbt-plugin;3.9.0 [warn] ==== typesafe-ivy-releases: tried [warn] http://repo.typesafe.com/typesafe/ivy-releases/com.twitter/scrooge-sbt-plugin/scala_2.10/sbt_0.13/3.9.0/ivys/ivy.xml [warn] ==== sbt-plugin-releases: tried [warn] http://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.twitter/scrooge-sbt-plugin/scala_2.10/sbt_0.13/3.9.0/ivys/ivy.xml [warn] ==== local: tried [warn] /Users/andreas/.ivy2/local/com.twitter/scrooge-sbt-plugin/scala_2.10/sbt_0.13/3.9.0/ivys/ivy.xml [warn] ==== public: tried [warn] http://repo1.maven.org/maven2/com/twitter/scrooge-sbt-plugin_2.10_0.13/3.9.0/scrooge-sbt-plugin-3.9.0.pom [warn] ==== sbt-plugin-releases: tried [warn] http://scalasbt.artifactoryonline.com/scalasbt/sbt-plugin-releases/com.twitter/scrooge-sbt-plugin/scala_2.10/sbt_0.13/3.9.0/ivys/ivy.xml [warn] ==== bintray-sbt-plugin-releases: tried [warn] http://dl.bintray.com/content/sbt/sbt-plugin-releases/com.twitter/scrooge-sbt-plugin/scala_2.10/sbt_0.13/3.9.0/ivys/ivy.xml [info] Resolving org.fusesource.jansi#jansi;1.4 ... [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] :: UNRESOLVED DEPENDENCIES :: [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] :: com.twitter#scrooge-sbt-plugin;3.9.0: not found [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] [warn] Note: Some unresolved dependencies have extra attributes. Check that these dependencies exist with the requested attributes. [warn] com.twitter:scrooge-sbt-plugin:3.9.0 (sbtVersion=0.13, scalaVersion=2.10) [warn] sbt.ResolveException: unresolved dependency: com.twitter#scrooge-sbt-plugin;3.9.0: not found

I am wondering how it could work for you? Did you compile scrooge yourself?

Thanks for looking into this!

[MasseGuillaume]

twitter publish everything to https://oss.sonatype.org/content/groups/public/com/twitter/

they did not build scrooge sbt plugin against sbt 0.13.

you can use this resolver in plugins.sbt:

resolvers += "CodeBrew-io" at http://codebrew-io.github.io/maven/

or build / publish the latest plugin to bintray

• create a bintray account
• git clone https://github.com/twitter/scrooge
• add bintray sbt plugin to the build definition
• project scrooge-sbt-plugin
• publish

[KenCoder]

I pushed akka-2.10 and also fixed the scala-2.10 branch

On Sun, Dec 22, 2013 at 10:04 PM, windemut notifications@github.com wrote:

There does not appear to be an akka-2.10 branch. I tried the scala2.10 branch again, and this time it started out with SBT set to scala-version = 2.10 (previously it was still set to 2.9.2). But, the issue with scrooge still appears to be there...

london:external andreas$ git clone -b scala2.10 https://github.com/KenCoder/scala-notebook-1 Cloning into 'scala-notebook-1'... remote: Counting objects: 1384, done. remote: Compressing objects: 100% (759/759), done. remote: Total 1384 (delta 380), reused 1307 (delta 317) Receiving objects: 100% (1384/1384), 702.84 KiB | 51.00 KiB/s, done. Resolving deltas: 100% (380/380), done. Checking connectivity... done london:external andreas$ cd scala-notebook-1/ london:scala-notebook-1 andreas$ sbt [info] Loading project definition from /Users/andreas/code/external/scala-notebook-1/project [warn] Multiple resolvers having different access mechanism configured with same name 'sbt-plugin-releases'. To avoid conflict, Remove duplicate project resolvers (resolvers) or rename publishing resolver (publishTo). [info] Updating {file:/Users/andreas/code/external/scala-notebook-1/project/}scala-notebook-1-build... [info] Resolving com.twitter#scrooge-sbt-plugin;3.9.0 ... [warn] module not found: com.twitter#scrooge-sbt-plugin;3.9.0 [warn] ==== typesafe-ivy-releases: tried [warn] http://repo.typesafe.com/typesafe/ivy-releases/com.twitter/scrooge-sbt-plugin/scala_2.10/sbt_0.13/3.9.0/ivys/ivy.xml [warn] ==== sbt-plugin-releases: tried [warn] http://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.twitter/scrooge-sbt-plugin/scala_2.10/sbt_0.13/3.9.0/ivys/ivy.xml [warn] ==== local: tried [warn] /Users/andreas/.ivy2/local/com.twitter/scrooge-sbt-plugin/scala_2.10/sbt_0.13/3.9.0/ivys/ivy.xml [warn] ==== public: tried [warn] http://repo1.maven.org/maven2/com/twitter/scrooge-sbt-plugin_2.10_0.13/3.9.0/scrooge-sbt-plugin-3.9.0.pom [warn] ==== sbt-plugin-releases: tried [warn] http://scalasbt.artifactoryonline.com/scalasbt/sbt-plugin-releases/com.twitter/scrooge-sbt-plugin/scala_2.10/sbt_0.13/3.9.0/ivys/ivy.xml [warn] ==== bintray-sbt-plugin-releases: tried [warn] http://dl.bintray.com/content/sbt/sbt-plugin-releases/com.twitter/scrooge-sbt-plugin/scala_2.10/sbt_0.13/3.9.0/ivys/ivy.xml [info] Resolving org.fusesource.jansi#jansi;1.4 ... [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] :: UNRESOLVED DEPENDENCIES :: [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] :: com.twitter#scrooge-sbt-plugin;3.9.0: not found [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] [warn] Note: Some unresolved dependencies have extra attributes. Check that these dependencies exist with the requested attributes. [warn] com.twitter:scrooge-sbt-plugin:3.9.0 (sbtVersion=0.13, scalaVersion=2.10) [warn] sbt.ResolveException: unresolved dependency: com.twitter#scrooge-sbt-plugin;3.9.0: not found

I am wondering how it could work for you? Did you compile scrooge yourself?

Thanks for looking into this!

— Reply to this email directly or view it on GitHubhttps://github.com/Bridgewater/scala-notebook/issues/2#issuecomment-31102529 .

[windemut]

Thanks, Ken! akka-2.10 worked for me. I now get the same error you mention, using sbt. I am afraid I won't be able to help with that. If there is still consideration of change of web framework, I can contribute my 2 cents: I have gone through Play, unfiltered, and now spray. I am partial to spray, for now. It is quite cool once you start to get the Directive thing, and it makes total asynchrony fairly easy.

[KenCoder]

Ok, thanks for looking at it, and thanks for the advice. Maybe I'll think about spray in the next incarnation.

On Sat, Dec 28, 2013 at 8:40 PM, windemut notifications@github.com wrote:

Thanks, Ken! akka-2.10 worked for me. I now get the same error you mention, using sbt. I am afraid I won't be able to help with that. If there is still consideration of change of web framework, I can contribute my 2 cents: I have gone through Play, unfiltered, and now spray. I am partial to spray, for now. It is quite cool once you start to get the Directive thing, and it makes total asynchrony fairly easy.

— Reply to this email directly or view it on GitHubhttps://github.com/Bridgewater/scala-notebook/issues/2#issuecomment-31309050 .

[paulp]

So I don't know what a lot of the references in this thread are, but not knowing about any of this work toward 2.10 I idly started porting it based on what's in master and it all seems to work, other than repl interruption (which you could probably recover from the scala repo history.) But I might not be the best judge of how well it works because I'd never looked at it before. In any case, #36.

[paulp]

...but requires a fix in scalamock more recent than the latest release.

[nightscape]

The ScalaMock dependency problem is fixed in Paul's PR. Can anyone merge his PR? :D

[Bridgewater]

Thanks Paul!