Cerfoglg
commented
8 years ago Let's look at each of those considerations:
- User friendliness: from my perspective, Python is a far more user friendly language in itself. It's less verbose than most languages, and it makes it easy to quickly write a script to submit to Spark. However, the ease of use of a language is ultimately very subjective, so whether or not someone chooses Python or Scala comes down to personal preference in this regard. I do personally like Python a lot, but I do realise that others may not agree, but most importantly I understand that there are far more important points to consider.
- Expressiveness of the language: ultimately, both Python and Scala are equally as expressive for what we need to do with Spark. What can be done with Scala can also be done with Python, and in either case you are still creating a spark context and calling functions on that (count, map, reduce, join, ...).
- Performance: Spark was written in Scala, so it goes without saying that Scala will have an advantage over Python in most cases. Where Python, or at least pypy, gives equal performances, as seen in the link you've provided, is when we are working with a lot of core, when now the language becomes irrelevant as the time required becomes bounded by other language independent factors. This is something to consider, because if we are working with a lot of cores then performance becomes less of a concern when choosing the language, and other factors might be more important to consider. Still, Scala has an advantage, and all evidence supports this fact.
- Data analysis support: it's true that Python has an edge thanks to some great data analysis libraries like Pandas, but it's also true that those libraries were designed to work on single systems rather than distributed ones. That's not to say that they can't come in handy and be of great use where Scala libraries may not perform as well. It's hard to know for sure until we actually start working with the analysers, getting a good idea as to what we actually have to do in our Spark scripts to compute the needed metrics.
One other thing that I think is worth noting is that we don't necessarely have to choose between only Python and only Scala. Since we have a microservice for sending scripts to Spark, it's possible to extend it to handle both Scala and Python scripts, and choose what language to use depending on the situation.
VincenzoFerme
We should carefully evaluate wether to use Python, as we are doing currently, or move to Scala.
The following consideration should be done:
Analyse PROs and CONs in a structured way in this issue.