search

probcomp / crosscat

A domain-general, Bayesian method for analyzing high-dimensional data tables
http://probcomp.csail.mit.edu/crosscat/
Apache License 2.0
322 stars 42 forks source link

Hadoop state non-functional #83

Open kadwanev opened 8 years ago

kadwanev commented 8 years ago

It looks like the HadoopEngine isn't currently in an executable state. There are references to settings that have been removed fairly recently. Is the implementation here something that could be made to work?

kadwanev commented 8 years ago

@gregory-marton question for you :)

gregory-marton commented 8 years ago

Sounds like you've done some investigation already. Can you share the details? In what way did it not work, which settings did you find, etc.?

kadwanev commented 8 years ago

Oh, I thought it was intentional.

https://github.com/probcomp/crosscat/blob/d7765df463e79055e5a9cf0031367c32e4a768d7/src/HadoopEngine.py#L27

There appears to not be a crosscat.settings since v0.1.25 The change that has broken seems sensible and straightforward but I would ask if the implementation, if fixed, would still be functional in the current version.

gregory-marton commented 8 years ago

I'm actually not sure of the history. I came onboard around v0.1.40, and this was not on my radar. @riastradh-probcomp if you have time, can you give more context?

@kadwanev, pull requests are always welcome!

riastradh-probcomp commented 8 years ago

Nobody has touched the Hadoop code in years, and it apparently requires various moving parts that were customized for one developer's setup years ago, with some private network layout and Amazon S3 account and local Hadoop installation &c.

I expect it would be easier to start from scratch than to try to revive what's there.

gregory-marton commented 8 years ago

Given that context, I expect the appropriate "fix" would be to remove HadoopEngine. @kadwanev, if you want to take this on instead, we would absolutely welcome it. If interested, let me know a time frame to check back with you?

kadwanev commented 8 years ago

Understood. Thanks for the responses.

Just want to ask: Is the distribution technique still sound? Did it ever work?

I ask this because the only reducer reference I see is /bin/cat, which leads me to question if it collected results back into a single response.

I want to know if the current implementation is a good starting point or not.

riastradh-probcomp commented 8 years ago

There is no 'reduce' step because Crosscat's job is just to apply a transition operator to each of a number of independent states -- it's all 'map', and it is embarrassingly parallelizable, so any parallelism you throw at it should stick, no matter how trivial.

The MultiprocessingEngine is just LocalEngine with Python map replaced by multiprocessing.pool().map to transition the states in separate processes. Doing the same on different computers will certainly work just fine.

kadwanev commented 8 years ago

Thanks. That definitely answers my question. I'll be looking to contribute some code as soon as possible.