Best practices for pipelines (Scala/Akka/Kafka/Spark)

[dazza-codes]

@Maatary is driving work on LD4P to leverage big-data pipelines for metadata wrangling. This includes working with scala, kafka and spark (and their dependencies, like zookeeper). We will need to document some developer and dev-ops best practices for these, including:

• scala
  • style guide and code static analysis
  • http://www.wartremover.org/ (like reek/rubocop for ruby)
  • http://www.scalastyle.org/
    • "Scalastyle is used as part of the grading framework for the course Functional Programming Principles in Scala by Martin Odersky on Coursera."
  • https://blog.codacy.com/review-of-scala-static-analysis-tools-11afa6451146
    • http://blog.codacy.com/2015/01/27/survey-of-scala-code-style/
    • https://github.com/databricks/scala-style-guide
    • https://github.com/paypal/scala-style-guide
    • https://github.com/alexandru/scala-best-practices
  • unit testing, functional testing
  • http://www.scalatest.org/
  • code coverage
  • configuration, e.g.
  • https://github.com/typesafehub/config
  • https://github.com/pureconfig/pureconfig
  • packaging and deployment
  • http://www.scala-sbt.org/
• kafka (zookeeper)
  • infrastructure configuration and provisioning
  • https://www.confluent.io/blog/design-and-deployment-considerations-for-deploying-apache-kafka-on-aws/
  • https://medium.com/netflix-techblog/kafka-inside-keystone-pipeline-dd5aeabaf6bb
  • application configuration (using shared_configs?)
  • monitoring
  • https://github.com/linkedin/kafka-monitor
• spark
  • infrastructure configuration and provisioning
  • connectivity to data source/sink services
  • application configuration (using shared_configs?)
• recommended reading
• example projects
  • Netflix
  • https://github.com/Netflix/suro (deprecated? can't find Keystone on github)
    • archived https://github.com/Netflix/suro/commit/e73d1a2e5b0492e41feb5a57d9e2a2e741a38453
  • https://medium.com/netflix-techblog/evolution-of-the-netflix-data-pipeline-da246ca36905
  • https://medium.com/netflix-techblog/kafka-inside-keystone-pipeline-dd5aeabaf6bb

Potential contributors: @Maatary @eefahy

[atz]

I think a lot of what we will be able to contribute here is "adopted conventions" in accordance w/ existing best practices principles.

My understanding, in the scope of the one project advancing this stack, about tools selected in some of the areas enumerated:

• sbt - Simple Build Tool (ivy/jar dependency mgmt, compiling)
• scalaTest - for testing Scala

[dazza-codes]

Best Practices in Scala Programming Guidelines for Ensuring a Professional Scala Code Base By Joshua Backfield OReilly - http://shop.oreilly.com/product/0636920051336.do

[dazza-codes]

@cmh2166 - tagging you to raise awareness of this as a deliverable for the LD4P pipeline sprint, with regard to the adoption and documentation of developer best practices.