Updated twitter_classifier Reference App to Spark 2 and Scala 2.11

mslinn commented 8 years ago

Summary of changes to the twitter_classifier reference app:

Updated from Spark 1.4 and Scala 2.10 to Spark 2 and Scala 2.11
Specified the SBT version as v0.13.13, previously no version of SBT had been specified
Rewrote 3 command-line programs so they look like Scala programs instead of Java programs.
Factored out common code for the 3 command-line programs
Provided scripts to drive the 3 command-line programs
Improved docs; still not quite happy with how the overall docs are laid out, might be good to discuss how to best bring it all together.

Although I am quite comfortable with Scala, this is my first time with Spark 2, so suggestions on how to best demonstrate the API would be most welcome.

mslinn commented 8 years ago

Thank you for the suggestion. I will remove master :)

Mike

mslinn commented 7 years ago

I wrote up a brief blog post on this new version of the Twitter Classifier Reference Application.

tashoyan commented 7 years ago

@mslinn, your post inspired me to write Scala implementation of Log Analyzer Application.

Commander Scala to parse command-line arguments
Scala ARM to avoid system resource leaks
Rich Scala collections API to prepare data for the report
Scala Option to safely operate with probably absent objects

Scala is awesome.

mslinn commented 7 years ago

@tashoyan Excellent! I'd be happy to do an idiomatic Scala style review, when ready. Let me know!

tashoyan commented 7 years ago

@mslinn My pull request is ready: https://github.com/databricks/reference-apps/pull/89 This is my first experience with Scala, so I would be grateful to you for a review.

databricks / reference-apps

Updated twitter_classifier Reference App to Spark 2 and Scala 2.11 #88