simleo
commented
6 years ago Hi, and thanks for reporting this.
Pydoop uses dynamic extension modules, so it's not importable from a zip archive. It should be importable from an egg (also supported by Spark), but this leads to the same error as above. I have just opened issue #276 for this and hope to get to it soon. In the meantime, since you most likely don't need properties anyway, you should be able to work around the problem as follows:
- Change
pydoop/__init__.pyso it does not break when properties are not found:
--- a/pydoop/__init__.py
+++ b/pydoop/__init__.py
@@ -179,9 +179,7 @@ def read_properties(fname):
with open(fname) as f:
parser.readfp(AddSectionWrapper(f))
except IOError as e:
- if e.errno != errno.ENOENT:
- raise
- return None # compile time, prop file is not there
+ return {}
return dict(parser.items(AddSectionWrapper.SEC_NAME))- Try the following (on a worker node, or a machine with the same configuration):
git clone --branch 1.2.0 https://github.com/crs4/pydoop
cd pydoop
export HADOOP_HOME=/your/hadoop/home
export JAVA_HOME=/your/java/home
python setup.py build
python setup.py bdist_eggYou should end up with a pydoop-1.2.0-py2.7.egg (or similar) under dist/. Try passing this file to sc.addPyFile instead of the zip one.
snalanagula
I have a requirement to write to hdfs inside map, hence am shipping pydoop.zip dependency module to all worker nodes using sc.addPyFile options, but when I try importing pydoop.hdfs I get below error.
Steps followed to create pydoop.zip
Sample pyspark code that I am trying to use pydoop inside map.
Please help me to resolve this.