search

hefnawi / dot-me

My Weblog Comments
https://hefnawi.me
1 stars 1 forks source link

https://hefnawi.me/posts/disco-erlang-python-mapreduce/ #3

Open utterances-bot opened 4 years ago

utterances-bot commented 4 years ago

Intro to Disco: an Erlang/Python MapReduce – Hefnawi

Prologue This post is my take on reviving an old project (last commit is 3 years ago) born around 2007⁄2008 at Nokia Research Center and written in Erlang. What was exciting for me is the fact that Disco project is capable of running Python MapReduce Jobs against an Erlang core, how awesome is that! — Erlang is a synonym for parallel processing and high availability. I successfully built it though and ran a 250M records dataset which is 10GB+ in size using a Python MapReduce job that finished in 28 minutes (improved from 44 minutes) using a cluster of 3 EC2 free-tier t2.

https://hefnawi.me/posts/disco-erlang-python-mapreduce/

kunthar commented 4 years ago

Great news o/ I have no time to give a hand but i'll follow your work for now.

hefnawi commented 4 years ago

Thanks a lot @kunthar Really appreciate your support =)