Open rmetzger opened 9 years ago
Were you able to get 2.3 running using bdutil ? I tried going Ambari 2.1 - HDP 2.3 route and it worked with some manual work at the end.
Changed the following in ambari.conf AMBARI_VERSION='2.1.0-1409' AMBARI_REPO="http://s3.amazonaws.com/dev.hortonworks.com/ambari/centos6/2.x/BUILDS/${AMBARI_VERSION}/ambaribn.repo" AMBARI_STACK='HDP' AMBARI_STACK_VERSION='2.3'
The install failed
Sun Jul 19 14:57:08 EDT 2015: Exited 1 : gcloud --project=iron-potion-771 --quiet --verbosity=info compute ssh hadoop-m --command=sudo su -l -c "cd ${PWD} && ./update-ambari-config.sh" 2>>update-ambari-config_deploy.stderr 1>>update-ambari-config_deploy.stdout --ssh-flag=-tt --ssh-flag=-oServerAliveInterval=60 --ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=us-central1-a Sun Jul 19 14:57:08 EDT 2015: Fetching on-VM logs from hadoop-m Warning: Permanently added '104.197.109.200' (RSA) to the list of known hosts. Sun Jul 19 14:57:11 EDT 2015: Command failed: wait ${SUBPROC} on line 326. Sun Jul 19 14:57:11 EDT 2015: Exit code of failed command: 1 Sun Jul 19 14:57:11 EDT 2015: Detailed debug info available in file: /tmp/bdutil-20150719-143917-aAi/debuginfo.txt Sun Jul 19 14:57:11 EDT 2015: Check console output for error messages and/or retry your command.
Error file ended with this ..
hadoop-m:tambari_wait status: INPROGRESS hadoop-m:tambari_wait status: INPROGRESS hadoop-m:tambari_wait status: INPROGRESS hadoop-m:tambari_wait status: INPROGRESS hadoop-m:t-test: Fatal internal error hadoop-m:tjava.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem not found hadoop-m:t at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195) hadoop-m:t at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2638) hadoop-m:t at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651) hadoop-m:t at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) hadoop-m:t at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) hadoop-m:t at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) hadoop-m:t at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) hadoop-m:t at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) hadoop-m:t at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325) hadoop-m:t at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235) hadoop-m:t at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218) hadoop-m:t at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201) hadoop-m:t at org.apache.hadoop.fs.shell.Command.run(Command.java:165) hadoop-m:t at org.apache.hadoop.fs.FsShell.run(FsShell.java:287) hadoop-m:t at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) hadoop-m:t at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) hadoop-m:t at org.apache.hadoop.fs.FsShell.main(FsShell.java:340) hadoop-m:tCaused by: java.lang.ClassNotFoundException: Class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem not found hadoop-m:t at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) hadoop-m:t at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193) hadoop-m:t ... 16 more hadoop-m:tError: There was a problem accessing your configuration bucket using the GCS hadoop-m:tconnector. Check configuration files. Also make sure have the GCS JSON API hadoop-m:tenabled as described at https://developers.google.com/storage/docs/json_api/. hadoop-m:t.
However the JSON API was fine .. but the hadoop-env.sh did not have the class path update to add location of the gcs connector jar.
Added the blow to hadoop env template in ambari config guy
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/usr/local/lib/hadoop/lib/gcs-connector-1.4.1-hadoop2.jar
Seems like everything is working but I guess I might need some more storage soon :)
hadoop fs -df -h gs://hadoop1213 15/07/19 19:48:22 INFO gcs.GoogleHadoopFileSystemBase: GHFS version: 1.4.1-hadoop2 Filesystem Size Used Available Use% gs://hadoop1213/ 8.0 E 0 8.0 E 0%
Were you guys able to run hdp 2.3 with the gdutil utility ??
can you please share the ambari configuration to do so ??
Hi, I'm trying to deploy HDP 2.3 with bdutil.
I've set these configuration values
in
ambari.conf
.The deployment fails with
/tmp/bdutil-20150629-102300-GFV/debuginfo.txt
contains:Does the hdp module in
bdutil
support HDP 2.3 ?