search

TresAmigosSD / SMV

Spark Modularized View
Apache License 2.0
42 stars 22 forks source link

Upgrade to Spark 2.1 #466

Closed Kai-Chen closed 7 years ago

Kai-Chen commented 7 years ago

The library asciigraph has currently no build for scala-2.11. New commits are being checked in, and there is a PR for 2.11 build, but it didn't get merged and is now failing.

Kai-Chen commented 7 years ago

@ninjapapa

Just noting that a stddev function is introduced in the o.a.s.sql.functions namespace, which collides with the stddev object in the o.t.smv namespace, defined in AggFuncs.scala as a UserDefinedAggregateFunction.

The port will continue to use the stddev in the smv namespace. But may need to see if the Spark implementation is stable enough (meaning without the quirks found in earlier first etc) to switch to.

Kai-Chen commented 7 years ago

All smv-related functions are ported after this commit, except for asciigraph, which doesn't have a build for scala 2.11. Also maven's pom.xml is not yet checked in.

This port is waiting to be merged into the main after a final release of the 1.5 line.

AliTajeldin commented 7 years ago

@Kai-Chen : let's also check-in a working pom.xml file as well. Until we make a complete move to sbt (#464) and find a way to either install or have an sbt stub(#507), most people will probably continue to use maven.

hubertp-lshift commented 7 years ago

Re: asciigraph - you might be better off forking the project and pulling it into your own repo. Given a recent change (https://github.com/mdr/ascii-graphs/commit/61fb705456983f9410ac14818472c1ae94ea3fb8) it looks to me that the maintainer is more likely to release 2.12.x, rather than 2.11.x. And the PR for 2.11.x has been there for 2 years.

Kai-Chen commented 7 years ago

Yes, we've considered that option and have decided to find a better way to visualize module dependency trees.

hubertp-lshift commented 7 years ago

@Kai-Chen Do you have a list of tasks that still need to be completed for this to move on? I've been playing around with 2.1 branch and would be happy to pick up sth if needed.

ninjapapa commented 7 years ago

We will highly likely to have a full review of the 2.1 branch this week, and tag a version, then move forward with full speed.

The key decision need to make is actually on what to do with the earlier versions.

ninjapapa commented 7 years ago

A new tag was created on the 2.1 branch: v2.1.0-prerelease-2. It passed all the tests and functions are in-sync with the v1.5.2.5 release, except the ascii graph.

For the next a couple of weeks, the master branch will still be on 1.5.2 and have some cleanups. Those cleanups will also get merged to the 2.1 branch. After about 2 weeks, we will shift the master to 2.1.

hubertp-lshift commented 7 years ago

I would be happy to fix all those warnings related to the upgrade. It looks pretty ugly when you do a clean compile.

jacobdr commented 7 years ago

👍

Jacob

On Mar 31, 2017, 08:40 -0400, hubertp-lshift notifications@github.com, wrote:

I would be happy to fix all those warnings related to the upgrade. It looks pretty ugly when you do a clean compile.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub (https://github.com/TresAmigosSD/SMV/issues/466#issuecomment-290701094), or mute the thread (https://github.com/notifications/unsubscribe-auth/AE8mM3Bom-31YwvjyeFKyuE1M-UR37ANks5rrPQhgaJpZM4L53lk).

ninjapapa commented 7 years ago

@hubertp-lshift Thanks for taking this! Please go ahead and do so. May need to do a pull request when done.

hubertp-lshift commented 7 years ago

@ninjapapa Sure. I will send one soon'ish. Also I noticed that this (or maybe the 1.x version as well?) branch has some memory leaks. Running 'test' in the sbt console will quickly build up used Metaspace. I will see if the profiler can point me to something quickly.