search

databricks / learning-spark

Example code from Learning Spark book
MIT License
3.89k stars 2.42k forks source link

Example 4-12 - PerKeyAvg for Python Incorrect #24

Open funseiki opened 8 years ago

funseiki commented 8 years ago

In the example, the map method shows to take a lambda with two parameters (key and xy), but it appears as though the python version of spark only has a map method that expects a lambda with just a single parameter.

So instead of the following

r = sumCount.map(lambda key, xy: (key, xy[0]/xy[1])).collectAsMap()

We should use

 r = sumCount.map( lambda kvp: ( kvp[0], kvp[1][0] / kvp[1][1] ) ).collectAsMap()
funseiki commented 8 years ago

Note: It appears the github source foregoes the mapping step and returns a list of {key: (sum, count)} instead of {key: avg}.