search

crs4 / pydoop

A Python MapReduce and HDFS API for Hadoop
Apache License 2.0
237 stars 59 forks source link

pydoop doesnot work for Hortonworks HDP #158

Closed sd2302 closed 9 years ago

sd2302 commented 9 years ago

So question: is it possible to port the hadoop_utils.py from CDH to HDP (hortonworks 2.2), please??

Do i need to submit a jira for porting pydoop for hortonworks ?

Here is the hadoop_home i used when manually build/insstall it, which works, but runtime didnot work but it cannot the the classes in hadoop_utils.py script

export HADOOP_HOME=/usr/hdp/2.2.0.0-2041/hadoop:/usr/hdp/2.2.0.0-2041/hadoop/conf:/usr/hdp/2.2.0.0-2041/hadoop/lib/:/usr/hdp/2.2.0.0-2041/hadoop/.//:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/./:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/.//:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/:/usr/hdp/2.2.0.0-2041/hadoop-yarn/.//:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/.//::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/java/mysql-connector-java.jar:/usr/share/java/ojdbc6.jar:/usr/hdp/current/hadoop-mapreduce-client/:/usr/hdp/current/tez-client/:/usr/hdp/current/tez-client/lib/:/etc/tez/conf/:/usr/hdp/2.2.0.0-2041/tez/:/usr/hdp/2.2.0.0-2041/tez/:/usr/hdp/2.2.0.0-2041/tez/lib/:/etc/tez/conf

test case:

Sucessfully installed pydoop but it doesnot work at all. Trying to run this simple script and get classLoad (loadFileSystems error:) error

Please help me.

        import pydoop
        print pydoop.hadoop_version()
        2.6.0.2.2.0.0-2041
        print pydoop.hadoop_classpath()
        /usr/hdp/current/hadoop-client/lib/native:/etc/hadoop/conf.empty

[hdfs@ham-dal-d001 sd2302]$ cat t.py import pydoop.hdfs as hdfs with hdfs.open('hdfs://ham-dal-d001.corp.wayport.net:8020/prod/hadoop/smallsite/clearpass_dev_login_1/login-1-dev.attwifi.com.messages.1') as f: for line in f:

do_something(line)

print line

hdfs@ham-dal-d001 sd2302]$ python t.py loadFileSystems error: (unable to get stack trace for java.lang.NoClassDefFoundError exception: ExceptionUtils::getStackTrace error.) hdfsBuilderConnect(forceNewInstance=0, nn=ham-dal-d001.corp.wayport.net, port=8020, kerbTicketCachePath=(NULL), userName=(NULL)) error: (unable to get stack trace for java.lang.NoClassDefFoundError exception: ExceptionUtils::getStackTrace error.) Traceback (most recent call last): File "t.py", line 2, in with hdfs.open('hdfs://ham-dal-d001.corp.wayport.net:8020/prod/hadoop/smallsite/clearpass_dev_login_1/login-1-dev.attwifi.com.messages.1') as f: File "/usr/lib/python2.6/site-packages/pydoop/hdfs/init.py", line 121, in open fs = hdfs(host, port, user) File "/usr/lib/python2.6/site-packages/pydoop/hdfs/fs.py", line 150, in init h, p, u, fs = _get_connection_info(host, port, user) File "/usr/lib/python2.6/site-packages/pydoop/hdfs/fs.py", line 64, in _get_connection_info fs = core_hdfs_fs(host, port, user) File "/usr/lib/python2.6/site-packages/pydoop/hdfs/core/init.py", line 57, in core_hdfs_fs return _CORE_MODULE.CoreHdfsFs(host, port, user) RuntimeError: (255, 'Unknown error 255')

elzaggo commented 9 years ago

HI. I believe that the develop branch has some support for hdp2.2, at least it should be able to detect it. Give it a try and let me know how it goes.

--gianluigi

sd2302 commented 9 years ago

I belive I'm already using the "develop" branch, yet, it doesn't work, same errors as originally reported, it doesnt pickup the hdp2.2 classpath b/c hadoop_util.py still has ONLY cloudera parcel references.

Please advise, thanks again elzaggo

[root@ham-dal-d001 pydoop]# pwd /usr/local/bin/pydoop [root@ham-dal-d001 pydoop]# git branch

mdrio commented 9 years ago

I think you are using a bad HADOOP_HOME. It must be simply the path where hadoop is installed. You have a list of paths instead.

Il giorno ven 14 ago 2015 19:51 sd2302 notifications@github.com ha scritto:

I belive I'm already using the "develop" branch, yet, it doesn't work, same errors as originally reported, it doesnt pickup the hdp2.2 classpath b/c hadoop_util.py still has ONLY cloudera parcel references.

Please advise, thanks again elzaggo

[root@ham-dal-d001 pydoop]# pwd /usr/local/bin/pydoop [root@ham-dal-d001 pydoop]# git branch

  • develop

— Reply to this email directly or view it on GitHub https://github.com/crs4/pydoop/issues/158#issuecomment-131194002.

elzaggo commented 9 years ago

Hi.

I setup a simple debian machine with hdp 2.2 using the appropriate .deb packages, git cloned pydoop and installed the version in develop. Things were configured to use an external hdfs and yarn cluster. All tests run with no problems -- apart from the chown and related tests since they will currently work only in pseudo-cluster mode. Basic pydoop submit examples worked fine too.

On the other hand, see below, I seem to be getting a much richer classpath that the one you are reporting. What happens if you try hadoop classpath ?

aen@client:~/pydoop/test/hdfs$ python  
Python 2.7.9 (default, Mar  1 2015, 12:57:24) 
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pydoop
>>> pydoop.hadoop_version()
2.6.0.2.2.0.0-2041
>>> pydoop.hadoop_classpath()
/usr/hdp/2.2.0.0-2041/hadoop/hadoop-auth.jar:/usr/hdp/2.2.0.0-2041/hadoop/hadoop-common.jar:/usr/hdp/2.2.0.0-2041/hadoop/hadoop-nfs-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop/hadoop-common-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop/hadoop-annotations.jar:/usr/hdp/2.2.0.0-2041/hadoop/hadoop-nfs.jar:/usr/hdp/2.2.0.0-2041/hadoop/hadoop-annotations-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop/hadoop-common-2.6.0.2.2.0.0-2041-tests.jar:/usr/hdp/2.2.0.0-2041/hadoop/hadoop-common-tests.jar:/usr/hdp/2.2.0.0-2041/hadoop/hadoop-auth-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jsch-0.1.42.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/asm-3.2.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/curator-client-2.6.0.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-el-1.0.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/ranger-hdfs-plugin-0.4.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-compress-1.4.1.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/httpcore-4.2.5.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-configuration-1.6.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jersey-json-1.9.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/avro-1.7.4.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/log4j-1.2.17.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/slf4j-log4j12-1.7.5.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/zookeeper-3.4.6.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-math3-3.1.1.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/curator-recipes-2.6.0.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/curator-framework-2.6.0.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/eclipselink-2.5.2-M1.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-codec-1.4.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-digester-1.8.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/stax-api-1.0-2.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/servlet-api-2.5.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-httpclient-3.1.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-logging-1.1.3.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/slf4j-api-1.7.5.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/ranger-plugins-audit-0.4.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-io-2.4.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/hamcrest-core-1.3.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jackson-core-2.2.3.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-lang-2.6.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jets3t-0.9.0.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jsr305-1.3.9.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/htrace-core-3.0.4.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/ranger-plugins-common-0.4.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/activation-1.1.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-collections-3.2.1.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/httpclient-4.2.5.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/junit-4.11.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/xz-1.0.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/javax.persistence-2.1.0.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-net-3.1.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/paranamer-2.3.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/ranger-plugins-impl-0.4.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jersey-server-1.9.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/mysql-connector-java.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/gson-2.2.4.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/guava-11.0.2.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/ojdbc6.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-cli-1.2.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/mockito-all-1.8.5.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jettison-1.1.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/netty-3.6.2.Final.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jersey-core-1.9.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/ranger-plugins-cred-0.4.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/hadoop-hdfs-2.6.0.2.2.0.0-2041-tests.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/hadoop-hdfs-nfs.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/hadoop-hdfs-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/hadoop-hdfs-nfs-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/hadoop-hdfs-tests.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/hadoop-hdfs.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/asm-3.2.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/commons-el-1.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/jsp-api-2.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/jsr305-1.3.9.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/htrace-core-3.0.4.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/jasper-runtime-5.5.23.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-server-nodemanager.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-server-nodemanager-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-common-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-server-common.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-server-tests-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-client.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-registry-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-api.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-common.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-server-web-proxy.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-server-tests.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-server-common-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-server-web-proxy-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-registry.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-client-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-server-resourcemanager-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-applications-distributedshell-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-api-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jsch-0.1.42.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/asm-3.2.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/javax.inject-1.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/curator-client-2.6.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/commons-el-1.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/httpcore-4.2.5.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/commons-configuration-1.6.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/avro-1.7.4.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/zookeeper-3.4.6.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jsp-api-2.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/commons-math3-3.1.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/curator-recipes-2.6.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/curator-framework-2.6.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jasper-compiler-5.5.23.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jline-0.9.94.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/commons-digester-1.8.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/commons-httpclient-3.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/commons-io-2.4.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jackson-core-2.2.3.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jets3t-0.9.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jsr305-1.3.9.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/htrace-core-3.0.4.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/activation-1.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/commons-collections-3.2.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/httpclient-4.2.5.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/xz-1.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/commons-net-3.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/xmlenc-0.52.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/paranamer-2.3.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/gson-2.2.4.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/guava-11.0.2.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jettison-1.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/guice-3.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/jasper-runtime-5.5.23.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-gridmix-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/jsch-0.1.42.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-client-core.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/api-util-1.0.0-M20.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-client-core-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-client-app.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-auth.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/asm-3.2.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-sls.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/curator-client-2.6.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/commons-el-1.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/commons-compress-1.4.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-ant-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/httpcore-4.2.5.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/jetty-6.1.26.hwx.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.6.0.2.2.0.0-2041-tests.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/commons-configuration-1.6.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/jersey-json-1.9.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/protobuf-java-2.5.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/avro-1.7.4.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-client-shuffle.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-archives-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/log4j-1.2.17.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/zookeeper-3.4.6.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/jsp-api-2.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-extras.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/snappy-java-1.0.4.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/commons-math3-3.1.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/curator-recipes-2.6.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/curator-framework-2.6.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-archives.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/jasper-compiler-5.5.23.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/java-xmlbuilder-0.4.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-client-common.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-streaming-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-client-jobclient.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/commons-beanutils-1.7.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-client-hs.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/commons-codec-1.4.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/commons-digester-1.8.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/stax-api-1.0-2.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-client-common-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/servlet-api-2.5.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-client-app-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-distcp.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/commons-httpclient-3.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/commons-logging-1.1.3.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/jaxb-api-2.2.2.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/commons-io-2.4.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/jackson-databind-2.2.3.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-examples-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hamcrest-core-1.3.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/jackson-core-2.2.3.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-openstack.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/commons-lang-2.6.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/jets3t-0.9.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-datajoin.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/jsr305-1.3.9.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/htrace-core-3.0.4.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/aws-java-sdk-1.7.4.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/metrics-core-3.0.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/activation-1.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/commons-collections-3.2.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-ant.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-extras-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/httpclient-4.2.5.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/junit-4.11.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/xz-1.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-client-shuffle-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/commons-net-3.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/xmlenc-0.52.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/paranamer-2.3.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/jackson-xc-1.9.13.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/jersey-server-1.9.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-sls-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-openstack-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-aws.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/gson-2.2.4.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/guava-11.0.2.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-datajoin-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-auth-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-client-hs-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-aws-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-examples.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/commons-cli-1.2.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/mockito-all-1.8.5.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-rumen-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/jackson-annotations-2.2.3.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-rumen.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-mapreduce-client-hs-plugins.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/joda-time-2.5.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/jettison-1.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/netty-3.6.2.Final.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-distcp-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/jersey-core-1.9.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-streaming.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/jackson-core-asl-1.9.13.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/hadoop-gridmix.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/jasper-runtime-5.5.23.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/asm-3.2.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/junit-4.11.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/xz-1.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/guice-3.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.2.0.0-2041/hadoop/hadoop/lib/native:/usr/hdp/2.2.0.0-2041/hadoop/etc/hadoop
tonyswu commented 9 years ago

I am unable to install pydoop with Hortonworks as well. The version we are using is 2.2.4.2, and I am using the following for $HADOOP_HOME:

export HADOOP_HOME=/usr/hdp/2.2.4.2-2/hadoop:/usr/hdp/2.2.4.2-2/hadoop/conf:/usr/hdp/2.2.4.2-2/hadoop/lib/:/usr/hdp/2.2.4.2-2/hadoop-hdfs/:/usr/hdp/2.2.4.2-2/hadoop-hdfs/lib/

But building gives the following errors:

using setuptools version 18.0.1 Traceback (most recent call last): File "setup.py", line 65, in import pydoop File "/home/tonywu/pydoop/pydoop/pydoop/init.py", line 42, in _HADOOP_INFO = _PATH_FINDER.find() # fill the cache ASAP File "/home/tonywu/pydoop/pydoop/pydoop/hadoop_utils.py", line 607, in find info[a] = getattr(self, a)() File "/home/tonywu/pydoop/pydoop/pydoop/hadoop_utils.py", line 422, in hadoop_home _hadoop_home_from_version_cmd() or File "/home/tonywu/pydoop/pydoop/pydoop/hadoop_utils.py", line 383, in _hadoop_home_from_version_cmd output = sp.check_output([hadoop_exec, 'version']) AttributeError: 'module' object has no attribute 'check_output'

Please advise.

elzaggo commented 9 years ago

Hi.

See https://github.com/crs4/pydoop/blob/develop/dev_tools/docker/client_side_tests/hdp_2.2.0.0/local_client_setup.sh

for an example of a script that will setup a docker container with pydoop and hdp2.2.

Let me know if this is useful.

--gianluigi

ilveroluca commented 9 years ago

That attribute error suggests you're using an unsupported version of python.

Luca Il 19/ago/2015 20:16, "Tony S. Wu" notifications@github.com ha scritto:

I am unable to install pydoop with Hortonworks as well. The version we are using is 2.2.4.2, and I am using the following for $HADOOP_HOME:

export HADOOP_HOME=/usr/hdp/2.2.4.2-2/hadoop:/usr/hdp/2.2.4.2-2/hadoop/conf:/usr/hdp/2.2.4.2-2/hadoop/lib/:/usr/hdp/2.2.4.2-2/hadoop-hdfs/:/usr/hdp/2.2.4.2-2/hadoop-hdfs/lib/

But building gives the following errors:

using setuptools version 18.0.1 Traceback (most recent call last): File "setup.py", line 65, in import pydoop File "/home/tonywu/pydoop/pydoop/pydoop/init.py", line 42, in _HADOOP_INFO = _PATH_FINDER.find() # fill the cache ASAP File "/home/tonywu/pydoop/pydoop/pydoop/hadoop_utils.py", line 607, in find info[a] = getattr(self, a)() File "/home/tonywu/pydoop/pydoop/pydoop/hadoop_utils.py", line 422, in hadoop_home _hadoop_home_from_version_cmd() or File "/home/tonywu/pydoop/pydoop/pydoop/hadoop_utils.py", line 383, in _hadoop_home_from_version_cmd output = sp.check_output([hadoop_exec, 'version']) AttributeError: 'module' object has no attribute 'check_output'

Please advise.

— Reply to this email directly or view it on GitHub https://github.com/crs4/pydoop/issues/158#issuecomment-132729513.

sd2302 commented 9 years ago

Hi Elzaggo,

it still doesnt work, yes, i do get a richer classpath as below..

[root@ham-dal-d001 sv164v]# export HADOOP_HOME=/usr/hdp/2.2.0.0-2041/hadoop [root@ham-dal-d001 sv164v]# export JAVA_HOME=/usr/jdk64/jdk1.7.0_67 [root@ham-dal-d001 sv164v]# export HADOOP_CONF_DIR=/etc/hadoop/conf

execfile('hdfs_test.py') Traceback (most recent call last): File "", line 1, in File "hdfs_test.py", line 2, in with hdfs.open('hdfs://ham-dal-d001.corp.wayport.net:8020/prod/hadoop/smallsite/clearpass_dev_login_1/login-1-dev.attwifi.com.messages.1') as f: File "/usr/lib/python2.6/site-packages/pydoop-1.0.0-py2.6.egg/pydoop/hdfs/init.py", line 121, in open fs = hdfs(host, port, user) File "/usr/lib/python2.6/site-packages/pydoop-1.0.0-py2.6.egg/pydoop/hdfs/fs.py", line 150, in init h, p, u, fs = _get_connection_info(host, port, user) File "/usr/lib/python2.6/site-packages/pydoop-1.0.0-py2.6.egg/pydoop/hdfs/fs.py", line 64, in _get_connection_info fs = core_hdfs_fs(host, port, user) File "/usr/lib/python2.6/site-packages/pydoop-1.0.0-py2.6.egg/pydoop/hdfs/core/init.py", line 55, in core_hdfs_fs 'module not initialized, check that Pydoop is correctly installed' RuntimeError: module not initialized, check that Pydoop is correctly installed import pydoop
pydoop.hadoop_classpath() '/usr/hdp/2.2.0.0-2041/hadoop/hadoop-auth.jar:/usr/hdp/2.2.0.0-2041/hadoop/hadoop-nfs.jar:/usr/hdp/2.2.0.0-2041/hadoop/hadoop-annotations-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop/hadoop-common.jar:/usr/hdp/2.2.0.0 -2041/hadoop/hadoop-auth-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop/hadoop-common-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop/hadoop-annotations.jar:/usr/hdp/2.2.0.0-2041/hadoop/hadoop-nfs-2.6.0.2.2.0.0-2041 .jar:/usr/hdp/2.2.0.0-2041/hadoop/hadoop-common-2.6.0.2.2.0.0-2041-tests.jar:/usr/hdp/2.2.0.0-2041/hadoop/hadoop-common-tests.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-collections-3.2.1.jar:/usr/hdp/2.2.0.0-2041/hadoop/li b/log4j-1.2.17.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/slf4j-api-1.7.5.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/ranger-plugins-impl-0.4.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/xz-1.0.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib /jsr305-1.3.9.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/servlet-api-2.5.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/guava-11.0.2.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-el-1.0.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/protobuf-java-2.5. 0.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/netty-3.6.2.Final.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/zookeep er-3.4.6.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/mockito-all-1.8.5.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jackson-ma pper-asl-1.9.13.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-lang-2.6.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/javax.persistence-2.1.0.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/stax-ap i-1.0-2.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jsch-0.1.42.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-io-2.4.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/ranger-plugins-co mmon-0.4.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/ranger-hdfs-plugin-0.4.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/curator-client-2.6.0.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/microsoft-windowsazure-storage -sdk-0.6.0.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-configuration-1.6.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jets3t-0.9.0.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-compress-1.4.1.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jet ty-util-6.1.26.hwx.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-cli-1.2.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-net-3.1.jar:/usr/hdp/2.2.0.0-2041/ha doop/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/paranamer-2.3.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-logging-1.1.3.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/curator-framework-2.6.0.jar:/usr/hdp/2.2.0.0-2041/had oop/lib/httpcore-4.2.5.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/mysql-connector-java.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jersey-server-1.9.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/htrace-core-3.0.4.jar:/usr/hdp/2.2.0.0-2041/hadoop/ lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/gson-2.2.4.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/ranger-plugins-audit-0.4.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-httpclient-3.1.jar:/usr/hdp /2.2.0.0-2041/hadoop/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/asm-3.2.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jettison-1.1.jar:/usr/hdp/2.2.0.0-2041/hadoop /lib/eclipselink-2.5.2-M1.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/ranger-plugins-cred-0.4.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/hamcrest-core-1.3.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jackson-xc-1.9.13.jar:/usr/hdp /2.2.0.0-2041/hadoop/lib/jackson-core-2.2.3.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/h dp/2.2.0.0-2041/hadoop/lib/commons-math3-3.1.1.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-digester-1.8.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/activation-1.1.jar:/usr /hdp/2.2.0.0-2041/hadoop/lib/jersey-core-1.9.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/slf4j-log4j12-1.7.5.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jersey-json-1.9.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/httpclient-4.2.5.jar:/usr/hdp/2. 2.0.0-2041/hadoop/lib/ojdbc6.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/junit-4.11.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/curator-recipes-2.6.0.jar:/usr/hdp/2.2.0.0-2041/hado op/lib/avro-1.7.4.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.2.0.0-2041/hadoop/lib/commons-codec-1.4.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/hadoop-hdfs-2.6.0.2.2.0.0-2041-tests.jar:/usr/hdp/2. 2.0.0-2041/hadoop-hdfs/hadoop-hdfs-nfs.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/hadoop-hdfs-2.6.0.2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/hadoop-hdfs-tests.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/hadoop-hdfs-nfs-2.6.0. 2.2.0.0-2041.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/hadoop-hdfs.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/jsr305-1.3.9.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/servlet-a pi-2.5.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/commons-el-1.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/x

elzaggo commented 9 years ago

Hi. Would it be possible for you to switch to python 2.7?

--gianluigi

elzaggo commented 9 years ago

We assume that switching to python 2.7 solved the issue.

lott3 commented 8 years ago

I got this Issue using HDP 2.4 and Python 2.7 --> http://stackoverflow.com/questions/37925300/pydoop-hdfs-ioexeption posted it on stackoverflow Someone got an idea how to solve it?

elzaggo commented 8 years ago

Hi. From the dump in stack-overflow, it looks more a problem with the path to the namenode. What happens if you put in the explicit hostname and port in the call to hdfs?

--gianluigi

lott3 commented 8 years ago

Same result if I explicit set hostname and port

ilveroluca commented 8 years ago

You should open a new issue rather than reviving old, closed issues.

You might have a different python interpreter being called in PySpark, or perhaps the same interpreter running with different environment variables? Open a new issue and tell us what's the relevant code you're running in PySpark.

Luca