search

crs4 / pydoop

A Python MapReduce and HDFS API for Hadoop
Apache License 2.0
236 stars 59 forks source link

Will pydoop support local mode? #330

Closed rickhg12hs closed 12 months ago

rickhg12hs commented 5 years ago

I wanted to use pydoop for some classwork, but I'm having trouble installing it.

pydoop.LocalModeNotSupported: ERROR: Hadoop is configured to run in local mode

Will/Does pydoop support local mode?

simleo commented 5 years ago

Hi,

we haven't looked at that bit in a while. Since we are in the middle of a big overhaul, there's a chance that that restriction does not even apply anymore (or can be lifted with a small amount of work). I will update this issue when I find the time to investigate.

All that being said, I'm not sure you really need local mode to play around and do some classwork. You can get a pre-configured pseudo-distributed Hadoop "cluster" with Pydoop by pulling our Docker image:

export PORT_FW="-p 8020:8020 -p 8042:8042 -p 8088:8088 -p 9000:9000 -p 10020:10020 -p 19888:19888 -p 9866:9866 -p 9867:9867 -p 9870:9870 -p 9864:9864 -p 9868:9868"
docker run --name pydoop ${PORT_FW} -d crs4/pydoop

Then visit http://localhost:9870, http://localhost:8088 and http://localhost:19888 for the web UIs, and/or get a shell on the container (to submit applications, etc.) with:

docker exec -it pydoop bash -l
rickhg12hs commented 5 years ago

Thanks for your response. Unfortunately, my classwork needs to run on the faculty's system. Apparently it's configured for local mode.

boiseamit commented 12 months ago

Hi!

I am trying to run the docker image on an Linux ARM (M1 pro) but it fails because the architecture doesn't match. Does this really need a new image?

ilveroluca commented 12 months ago

Please don't hijack existing issues with off-topic requests. Nevertheless, we've never generated images for ARM, so you'll need to build one yourself.