Spark 2: Upgrade Scala sources of Logs Analyzer application to Spark 2 and Scala 2.11

databricks / reference-apps

Spark reference applications

Other

656 stars 342 forks source link

Spark 2: Upgrade Scala sources of Logs Analyzer application to Spark 2 and Scala 2.11 #89

Closed tashoyan closed 7 years ago

tashoyan commented 7 years ago

Chapters 1, 2, 3 as well as App. Some implementation details on Scala code:

Commander Scala to parse command-line arguments
Scala ARM to avoid system resource leaks
Rich Scala collections API to prepare data for the report
Scala Option to safely operate with probably absent objects

Bonus: some clean-up in Java 8 examples.

bllchmbrs commented 7 years ago

Can you let me know when you're ready for me to take a look.

tashoyan commented 7 years ago

@anabranch, I am ready for your review.

tashoyan commented 7 years ago

@mslinn I addressed almost all your comments. There is a couple of them that I didn't get, so I asked my questions above.

mslinn commented 7 years ago

Don't forget to add project/build.properties and inside it, place:

sbt.version=0.13.13

tashoyan commented 7 years ago

Done. Thank you Mike for your careful review. I am pleased to work in this cooperative way. @anabranch I believe we are ready for your final check and merge.

bllchmbrs commented 7 years ago

I will check this out this week.

bllchmbrs commented 7 years ago

This is a huge pull request that unfortunately I don't have a lot of time to commit to right now. I'll trust you all and mention you if there's a bug someone finds :). Thanks for updating this :)