search

GoogleCloudDataproc / bdutil

[DEPRECATED] Script used to manage Hadoop and Spark instances on Google Compute Engine
https://cloud.google.com/dataproc
Apache License 2.0
109 stars 94 forks source link

bdutil fails to build hdp #65

Open lvnilesh opened 9 years ago

lvnilesh commented 9 years ago 
gsutil md gs://cloudgenius

bdutil -p beacloudgenius -b cloudgenius -P cg -z us-central1-f -e /home/user/bdutil/platforms/hdp/ambari_env.sh generate_config my.sh

cat my.sh

import_env /home/user/bdutil/platforms/hdp/ambari_env.sh
PROJECT=beacloudgenius
CONFIGBUCKET=cloudgenius
PREFIX=cg
GCE_ZONE=us-central1-f
GCE_MASTER_MACHINE_TYPE=n1-standard-4

bdutil -e my.sh deploy
Tue Sep 22 18:32:19 PDT 2015: Using local tmp dir for staging files: /tmp/bdutil-20150922-183219-iBw
Tue Sep 22 18:32:19 PDT 2015: Using custom environment-variable file(s): bdutil_env.sh my.sh
Tue Sep 22 18:32:19 PDT 2015: Reading environment-variable file: /home/user/bdutil/bdutil_env.sh
Tue Sep 22 18:32:19 PDT 2015: Reading environment-variable file: ./my.sh
Importing dependent env file: /home/user/bdutil/platforms/hdp/ambari_env.sh
Importing dependent env file: /home/user/bdutil/platforms/hdp/ambari_manual_env.sh
Importing dependent env file: /home/user/bdutil/hadoop2_env.sh
Importing dependent env file: /home/user/bdutil/platforms/hdp/ambari.conf
Importing dependent env file: /home/user/bdutil/platforms/hdp/ambari_functions.sh
Deploy cluster with following settings?
      CONFIGBUCKET='cloudgenius'
      PROJECT='beacloudgenius'
      GCE_IMAGE='centos-6'
      GCE_ZONE='us-central1-f'
      GCE_NETWORK='default'
      PREEMPTIBLE_FRACTION=0.0
      PREFIX='cg'
      NUM_WORKERS=4
      MASTER_HOSTNAME='cg-m'
      WORKERS='cg-w-0 cg-w-1 cg-w-2 cg-w-3'
      BDUTIL_GCS_STAGING_DIR='gs://cloudgenius/bdutil-staging/cg-m'
            MASTER_ATTACHED_PD='cg-m-pd'
      WORKER_ATTACHED_PDS='cg-w-0-pd cg-w-1-pd cg-w-2-pd cg-w-3-pd'
      (y/n) y
Tue Sep 22 18:32:21 PDT 2015: Checking for existence of gs://cloudgenius...
gs://cloudgenius/
Tue Sep 22 18:32:21 PDT 2015: Checking for existence of gs://hadoop-dist/hadoop-2.7.1.tar.gz...
Tue Sep 22 18:32:22 PDT 2015: Checking upload files...
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/conf/hadoop2/core-template.xml'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/conf/hadoop2/mapred-template.xml'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/conf/hadoop2/yarn-template.xml'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/conf/hadoop2/gcs-core-template.xml'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/conf/hadoop2/bigtable-hbase-site-template.xml'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/conf/hadoop2/hdfs-template.xml'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/conf/hadoop2/bq-mapred-template.xml'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/libexec/hadoop_helpers.sh'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/libexec/configure_mrv2_mem.py'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/hadoop2_env.sh'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/platforms/hdp/ambari.conf'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/platforms/hdp/ambari_functions.sh'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/libexec/hadoop_helpers.sh'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/platforms/hdp/configuration.json'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/platforms/hdp/resources/public-hostname-gcloud.sh'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/platforms/hdp/resources/thp-disable.sh'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/platforms/hdp/ambari_manual_env.sh'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/platforms/hdp/create_blueprint.py'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/platforms/hdp/ambari_env.sh'
Tue Sep 22 18:32:22 PDT 2015: Generating 12 command groups...
Tue Sep 22 18:32:22 PDT 2015: Done generating remote shell scripts.
Tue Sep 22 18:32:22 PDT 2015: Creating attached worker disks: cg-w-0-pd cg-w-1-pd cg-w-2-pd cg-w-3-pd
....Tue Sep 22 18:32:23 PDT 2015: Creating attached master disk: cg-m-pd
.Tue Sep 22 18:32:23 PDT 2015: Done creating disks!
Tue Sep 22 18:32:23 PDT 2015: Waiting on async 'disks create' jobs to finish. Might take a while...
.....
Tue Sep 22 18:32:32 PDT 2015: Creating worker instances: cg-w-0 cg-w-1 cg-w-2 cg-w-3
....Tue Sep 22 18:32:32 PDT 2015: Creating master instance: cg-m
.Tue Sep 22 18:32:33 PDT 2015: Waiting on async 'instances create' jobs to finish. Might take a while...
.....
Tue Sep 22 18:33:30 PDT 2015: Instances all created. Entering polling loop to wait for ssh-ability
.....Tue Sep 22 18:33:31 PDT 2015: Waiting on async 'wait_for_ssh' jobs to finish. Might take a while...
.....
Tue Sep 22 18:33:34 PDT 2015: Instances all ssh-able
Tue Sep 22 18:33:34 PDT 2015: Staging file and script dependencies into gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/...
Copying file:///home/user/bdutil/conf/hadoop2/mapred-template.xml [Content-Type=application/xml]...
Copying file:///home/user/bdutil/conf/hadoop2/core-template.xml [Content-Type=application/xml]...
Copying file:///home/user/bdutil/conf/hadoop2/yarn-template.xml [Content-Type=application/xml]...
Copying file:///home/user/bdutil/conf/hadoop2/gcs-core-template.xml [Content-Type=application/xml]...
Copying file:///home/user/bdutil/conf/hadoop2/bigtable-hbase-site-template.xml [Content-Type=application/xml]...
Copying file:///home/user/bdutil/conf/hadoop2/hdfs-template.xml [Content-Type=application/xml]...
Copying file:///home/user/bdutil/conf/hadoop2/bq-mapred-template.xml [Content-Type=application/xml]...
Uploading   ...aging/cg-m/20150922-183219-iBw/core-template.xml: 749 B/749 B    
Copying file:///home/user/bdutil/libexec/hadoop_helpers.sh [Content-Type=text/x-sh]...
Uploading   ...ing/cg-m/20150922-183219-iBw/mapred-template.xml: 5.43 KiB/5.43 KiB    
Uploading   ...aging/cg-m/20150922-183219-iBw/yarn-template.xml: 2.81 KiB/2.81 KiB    
Copying file:///home/user/bdutil/libexec/configure_mrv2_mem.py [Content-Type=text/x-python]...
Copying file:///home/user/bdutil/hadoop2_env.sh [Content-Type=text/x-sh]...
Uploading   ...g/cg-m/20150922-183219-iBw/gcs-core-template.xml: 2.62 KiB/2.62 KiB    
Uploading   ...0922-183219-iBw/bigtable-hbase-site-template.xml: 1.35 KiB/1.35 KiB    
Uploading   ...aging/cg-m/20150922-183219-iBw/hdfs-template.xml: 2.72 KiB/2.72 KiB    
Uploading   .../cg-m/20150922-183219-iBw/bq-mapred-template.xml: 769 B/769 B     
Uploading   ...aging/cg-m/20150922-183219-iBw/hadoop_helpers.sh: 5.98 KiB/5.98 KiB    
Uploading   ...-staging/cg-m/20150922-183219-iBw/hadoop2_env.sh: 2.92 KiB/2.92 KiB    
Uploading   ...g/cg-m/20150922-183219-iBw/configure_mrv2_mem.py: 7.28 KiB/7.28 KiB    
Copying file:///home/user/bdutil/platforms/hdp/ambari.conf [Content-Type=application/octet-stream]...
Copying file:///home/user/bdutil/platforms/hdp/ambari_functions.sh [Content-Type=text/x-sh]...
Copying file:///home/user/bdutil/libexec/hadoop_helpers.sh [Content-Type=text/x-sh]...
Copying file:///home/user/bdutil/platforms/hdp/configuration.json [Content-Type=application/json]...
Copying file:///home/user/bdutil/platforms/hdp/resources/public-hostname-gcloud.sh [Content-Type=text/x-sh]...
Uploading   ...til-staging/cg-m/20150922-183219-iBw/ambari.conf: 2.56 KiB/2.56 KiB    
Copying file:///home/user/bdutil/platforms/hdp/resources/thp-disable.sh [Content-Type=text/x-sh]...
Copying file:///home/user/bdutil/platforms/hdp/ambari_manual_env.sh [Content-Type=text/x-sh]...
Copying file:///home/user/bdutil/platforms/hdp/create_blueprint.py [Content-Type=text/x-python]...
Copying file:///home/user/bdutil/platforms/hdp/ambari_env.sh [Content-Type=text/x-sh]...
Copying file:///tmp/bdutil-20150922-183219-iBw/hadoop-env-setup.sh [Content-Type=text/x-sh]...
Uploading   ...ing/cg-m/20150922-183219-iBw/ambari_functions.sh: 3.62 KiB/3.62 KiB    
Uploading   ...aging/cg-m/20150922-183219-iBw/hadoop_helpers.sh: 5.98 KiB/5.98 KiB    
Uploading   ...-m/20150922-183219-iBw/public-hostname-gcloud.sh: 174 B/174 B    
Uploading   ...ging/cg-m/20150922-183219-iBw/configuration.json: 863 B/863 B    
Uploading   ...ing/cg-m/20150922-183219-iBw/hadoop-env-setup.sh: 30.36 KiB/30.36 KiB    
Uploading   ...-staging/cg-m/20150922-183219-iBw/thp-disable.sh: 451 B/451 B    
Uploading   ...l-staging/cg-m/20150922-183219-iBw/ambari_env.sh: 1.82 KiB/1.82 KiB    
Uploading   ...ing/cg-m/20150922-183219-iBw/create_blueprint.py: 4.12 KiB/4.12 KiB    
Uploading   ...ng/cg-m/20150922-183219-iBw/ambari_manual_env.sh: 3.47 KiB/3.47 KiB    
Copying file:///tmp/bdutil-20150922-183219-iBw/deploy-ssh-master-setup.sh [Content-Type=text/x-sh]...
Copying file:///tmp/bdutil-20150922-183219-iBw/deploy-core-setup.sh [Content-Type=text/x-sh]...
Copying file:///tmp/bdutil-20150922-183219-iBw/deploy-master-nfs-setup.sh [Content-Type=text/x-sh]...
Copying file:///tmp/bdutil-20150922-183219-iBw/deploy-client-nfs-setup.sh [Content-Type=text/x-sh]...
Copying file:///tmp/bdutil-20150922-183219-iBw/deploy-ssh-worker-setup.sh [Content-Type=text/x-sh]...
Copying file:///tmp/bdutil-20150922-183219-iBw/deploy-start.sh [Content-Type=text/x-sh]...
Copying file:///tmp/bdutil-20150922-183219-iBw/install_connectors.sh [Content-Type=text/x-sh]...
Copying file:///tmp/bdutil-20150922-183219-iBw/deploy_start2.sh [Content-Type=text/x-sh]...
Copying file:///tmp/bdutil-20150922-183219-iBw/ambari-setup.sh [Content-Type=text/x-sh]...
Uploading   ...m/20150922-183219-iBw/deploy-ssh-master-setup.sh: 1.59 KiB/1.59 KiB    
Uploading   ...m/20150922-183219-iBw/deploy-client-nfs-setup.sh: 1.53 KiB/1.53 KiB    
Uploading   ...staging/cg-m/20150922-183219-iBw/deploy-start.sh: 1.27 KiB/1.27 KiB    
Uploading   ...m/20150922-183219-iBw/deploy-master-nfs-setup.sh: 3.84 KiB/3.84 KiB    
Uploading   ...m/20150922-183219-iBw/deploy-ssh-worker-setup.sh: 1.37 KiB/1.37 KiB    
Uploading   ...staging/cg-m/20150922-183219-iBw/ambari-setup.sh: 9.22 KiB/9.22 KiB    
Uploading   ...taging/cg-m/20150922-183219-iBw/deploy_start2.sh: 1.41 KiB/1.41 KiB    
Uploading   ...g/cg-m/20150922-183219-iBw/install_connectors.sh: 6.19 KiB/6.19 KiB    
Uploading   ...ng/cg-m/20150922-183219-iBw/deploy-core-setup.sh: 28.41 KiB/28.41 KiB    
Copying file:///tmp/bdutil-20150922-183219-iBw/install-gcs-connector-on-ambari.sh [Content-Type=text/x-sh]...
Uploading   ...22-183219-iBw/install-gcs-connector-on-ambari.sh: 1.47 KiB/1.47 KiB    
Copying file:///tmp/bdutil-20150922-183219-iBw/install-ambari-components.sh [Content-Type=text/x-sh]...
Copying file:///tmp/bdutil-20150922-183219-iBw/update-ambari-config.sh [Content-Type=text/x-sh]...
Uploading   ...20150922-183219-iBw/install-ambari-components.sh: 3.9 KiB/3.9 KiB    
Uploading   ...cg-m/20150922-183219-iBw/update-ambari-config.sh: 2.88 KiB/2.88 KiB    
Tue Sep 22 18:33:36 PDT 2015: Downloading staging files onto VMs...
Tue Sep 22 18:33:36 PDT 2015: Invoking on workers: gcloud --quiet components update gsutil; gsutil -m cp gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/core-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/mapred-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/yarn-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/gcs-core-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/bigtable-hbase-site-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hdfs-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/bq-mapred-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop_helpers.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/configure_mrv2_mem.py gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop2_env.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari.conf gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari_functions.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop_helpers.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/configuration.json gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/public-hostname-gcloud.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/thp-disable.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari_manual_env.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/create_blueprint.py gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari_env.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop-env-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-ssh-master-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-core-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-master-nfs-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-client-nfs-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-ssh-worker-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-start.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/install_connectors.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy_start2.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/install-gcs-connector-on-ambari.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/update-ambari-config.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/install-ambari-components.sh . && chmod 755 *
....Tue Sep 22 18:33:37 PDT 2015: Invoking on master: gcloud --quiet components update gsutil; gsutil -m cp gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/core-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/mapred-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/yarn-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/gcs-core-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/bigtable-hbase-site-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hdfs-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/bq-mapred-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop_helpers.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/configure_mrv2_mem.py gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop2_env.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari.conf gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari_functions.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop_helpers.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/configuration.json gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/public-hostname-gcloud.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/thp-disable.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari_manual_env.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/create_blueprint.py gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari_env.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop-env-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-ssh-master-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-core-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-master-nfs-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-client-nfs-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-ssh-worker-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-start.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/install_connectors.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy_start2.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/install-gcs-connector-on-ambari.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/update-ambari-config.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/install-ambari-components.sh . && chmod 755 *
.Tue Sep 22 18:33:37 PDT 2015: Waiting on async 'ssh' jobs to finish. Might take a while...
.....
Tue Sep 22 18:33:51 PDT 2015: Uploads of shell scripts finished, deleting staging files...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/core-template.xml...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/mapred-template.xml...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/yarn-template.xml...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/gcs-core-template.xml...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/bigtable-hbase-site-template.xml...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hdfs-template.xml...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/bq-mapred-template.xml...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop_helpers.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/configure_mrv2_mem.py...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop2_env.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari.conf...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari_functions.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop_helpers.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/configuration.json...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/public-hostname-gcloud.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/thp-disable.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari_manual_env.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/create_blueprint.py...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari_env.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop-env-setup.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-ssh-master-setup.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-core-setup.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-master-nfs-setup.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-client-nfs-setup.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-ssh-worker-setup.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-start.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/install_connectors.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy_start2.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari-setup.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/install-gcs-connector-on-ambari.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/update-ambari-config.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/install-ambari-components.sh...
CommandException: 1 files/objects could not be removed.
Tue Sep 22 18:33:55 PDT 2015: Warning. Failed to delete all staging files.
Tue Sep 22 18:33:55 PDT 2015: Invoking on workers: ./ambari-setup.sh
....Tue Sep 22 18:33:56 PDT 2015: Invoking on master: ./ambari-setup.sh
.Tue Sep 22 18:33:56 PDT 2015: Waiting on async 'ssh' jobs to finish. Might take a while...
.....
Tue Sep 22 18:35:34 PDT 2015: Step 'ambari-setup,ambari-setup' done...
Tue Sep 22 18:35:34 PDT 2015: Invoking on master: ./deploy-master-nfs-setup.sh
.Tue Sep 22 18:35:34 PDT 2015: Waiting on async 'ssh' jobs to finish. Might take a while...
.
Tue Sep 22 18:35:41 PDT 2015: Step 'deploy-master-nfs-setup,*' done...
Tue Sep 22 18:35:41 PDT 2015: Invoking on workers: ./deploy-client-nfs-setup.sh
....Tue Sep 22 18:35:42 PDT 2015: Invoking on master: ./deploy-client-nfs-setup.sh
.Tue Sep 22 18:35:42 PDT 2015: Waiting on async 'ssh' jobs to finish. Might take a while...
.....
Tue Sep 22 18:35:55 PDT 2015: Step 'deploy-client-nfs-setup,deploy-client-nfs-setup' done...
Tue Sep 22 18:35:55 PDT 2015: Invoking on master: ./install-ambari-components.sh
.Tue Sep 22 18:35:55 PDT 2015: Waiting on async 'ssh' jobs to finish. Might take a while...
Tue Sep 22 19:36:14 PDT 2015: Exited 1 : gcloud --project=beacloudgenius --quiet --verbosity=info compute ssh cg-m --command=sudo su -l -c "cd ${PWD} && ./install-ambari-components.sh" 2>>install-ambari-components_deploy.stderr 1>>install-ambari-components_deploy.stdout --ssh-flag=-tt --ssh-flag=-oServerAliveInterval=60 --ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=us-central1-f
Tue Sep 22 19:36:14 PDT 2015: Fetching on-VM logs from cg-m
Warning: Permanently added '104.197.16.133' (RSA) to the list of known hosts.
Tue Sep 22 19:36:16 PDT 2015: Command failed: wait ${SUBPROC} on line 326.
Tue Sep 22 19:36:16 PDT 2015: Exit code of failed command: 1
Tue Sep 22 19:36:16 PDT 2015: Detailed debug info available in file: /tmp/bdutil-20150922-183219-iBw/debuginfo.txt
Tue Sep 22 19:36:16 PDT 2015: Check console output for error messages and/or retry your command.
(foobar)
lvnilesh commented 9 years ago

/tmp/bdutil-20150922-183219-iBw/debuginfo.txt is in this gist

https://gist.github.com/lvnilesh/bafce5637e31efb8dd68

dennishuo commented 9 years ago

Sorry for the delay, must have missed the notification. Did you hit this error consistently, or was it just once? If you try again and hit the error, can you try running the SSH-tunnel commands on https://github.com/GoogleCloudPlatform/bdutil/tree/master/platforms/hdp and then visiting master-hostname:8080 through a browser with username/password 'admin/admin' and seeing if Ambari's UI reports any errors?