StatCan / aaw-contrib-jupyter-notebooks

Jupyter Notebooks to be used with Advanced Analytics Workspace platform
Other
10 stars 13 forks source link

Adding a simple map-reduce kf-pipeline #4

Closed blairdrummond closed 4 years ago

blairdrummond commented 4 years ago

Super easy, it shows how to create a map image and a reduce image; run everything, pass json in/out, and plop all data in the minio bucket.

It's an MC estimate for pi. Every map container grabs a (uniform) random point in [-1,1]^2, and then returns 4 if the point is in the unit circle and 0 otherwise. The reduce step is just an average.

I think map-reduce is the simplest useful kf-pipeline, so thought it'd be nice to have an example.

You can take a look at the storage strategy and let me know if there's a better route.