Memory estimation for container of mapper/reducer

twitter / scalding

A Scala API for Cascading

http://twitter.com/scalding

Apache License 2.0

3.5k stars 706 forks source link

Memory estimation for container of mapper/reducer #1667

Closed dieu closed 7 years ago

dieu commented 7 years ago

I thinking to move reducer_estimation package to estimation.reducer and mark classes deprecated in reducer_estimation package until 0.19.0.

what do you think? //cc @piyushnarang @isnotinvain

Todo:

[x] Tests for memory estimator

johnynek commented 7 years ago

is there any way we can not commit a 20k like testing file? Maybe I'm being obsessive, but I'm not crazy about that. Can we do with a smaller test file?

dieu commented 7 years ago

@johnynek I'm also not a big fan of it. But I want to use real answers from hRaven, let's see what we can do:

flowResponse -- I don't want to truncate this file because it has pretty much everything that needs and after removing something we will put some assumption into the file.
jobResponses -- I will reduce the count of tasks.

Can you look again and says it's good enough reduction for you, please.

dieu commented 7 years ago

@johnynek can we move forward with this PR?

piyushnarang commented 7 years ago

@johnynek / @dieu can we merge this PR?