Define a Functional Testing Methodology for the scripts

[VincenzoFerme]

We need to define a functional testing methodology for the scripts, both for the python side and the Spark side.

Some references to start with:

• Python:
  • http://docs.python-guide.org/en/latest/writing/tests/
• Spark:
  • http://www.slideshare.net/SparkSummit/continuous-integration-for-spark-apps-by-sean-mcintyre (video: https://www.youtube.com/watch?v=NgQM_dppQxU&feature=youtu.be)
  • http://www.slideshare.net/hkarau/effective-testing-for-spark-programs-strata-ny-2015
  • https://spark-summit.org/2014/wp-content/uploads/2014/06/Testing-Spark-Best-Practices-Anupama-Shetty-Neil-Marshall.pdf
  • https://sparktutorials.github.io/2015/07/30/spark-testing-unit.html, https://sparktutorials.github.io/2015/09/27/spark-testing-functional.html
  • https://chapeau.freevariable.com/2015/04/spark-apps-and-ci.html
  • http://blog.uncharted.software/2015/11/continuous-integration-with-apache-spark/

[VincenzoFerme]

Defined in #56