search

cloudfoundry / bosh-bootloader

Command line utility for standing up a BOSH director on an IAAS of your choice.
Apache License 2.0
178 stars 180 forks source link

Unable to ssh into Bosh director #57

Closed ChunyiLyu closed 7 years ago

ChunyiLyu commented 7 years ago

We are using an environment that was setup with bbl on AWS. It was created in concourse, and is running our CI. We recently tried to ssh into the director using the bbl credentials which were checked into our repo as part of the CI in the following way:

++ pwd
+ DIR=/tmp/build/be72c34d
+ trap commit_bbl_state_file EXIT
+ pushd env-repo
/tmp/build/be72c34d/env-repo /tmp/build/be72c34d
+ set +x
bbl     --state-dir bbl-infrastructure     --aws-access-key-id [REDACTED]     --aws-secret-access-key [REDACTED]     --aws-region us-east-1     unsupported-deploy-bosh-on-aws-for-concourse
step: using existing keypair
step: generating cloudformation template
step: updating cloudformation stack
step: finished applying cloudformation template
step: generating bosh-init manifest
step: deploying bosh director
Deployment manifest: '/tmp/bosh-init283134969/bosh.yml'
Deployment state: '/tmp/bosh-init283134969/bosh-state.json'

Started validating
  Downloading release 'bosh'... Finished (00:00:03)
  Validating release 'bosh'... Finished (00:00:09)
  Downloading release 'bosh-aws-cpi'... Finished (00:00:00)
  Validating release 'bosh-aws-cpi'... Finished (00:00:00)
  Validating cpi release... Finished (00:00:00)
  Validating deployment manifest... Finished (00:00:00)
  Downloading stemcell... Finished (00:00:00)
  Validating stemcell... Finished (00:00:00)
Finished validating (00:00:14)
No deployment, stemcell or release changes. Skipping deploy.
step: generating cloud config
step: applying cloud config
+ bbl --state-dir bbl-infrastructure unsupported-create-lbs --type=cf --cert=/tmp/bbl-cert --key=/tmp/bbl-key --skip-if-exists
lb type "cf" exists, skipping...
+ bbl --state-dir bbl-infrastructure unsupported-update-lbs --cert=/tmp/bbl-cert --key=/tmp/bbl-key
no updates are to be performed
+ popd
/tmp/build/be72c34d
+ commit_bbl_state_file
+ pushd /tmp/build/be72c34d/env-repo
/tmp/build/be72c34d/env-repo /tmp/build/be72c34d
++ git status --porcelain
+ [[ -n '' ]]
+ popd
/tmp/build/be72c34d
+ pushd /tmp/build/be72c34d
/tmp/build/be72c34d /tmp/build/be72c34d
+ shopt -s dotglob
+ cp -R env-repo/.git env-repo/bbl-infrastructure env-repo/cf env-repo/diego updated-env-repo/
+ popd
/tmp/build/be72c34d

Our credentials were rejected and we were unable to debug our problem on the director:

ssh -vvv -i /tmp/hagrid-key root@52.206.238.7
OpenSSH_6.9p1, LibreSSL 2.1.8
debug1: Reading configuration data /Users/pivotal/.ssh/config
debug1: /Users/pivotal/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 52.206.238.7 [52.206.238.7] port 22.
debug1: Connection established.
debug1: identity file /tmp/hagrid-key type 1
debug1: key_load_public: No such file or directory
debug1: identity file /tmp/hagrid-key-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7 pat OpenSSH_6.6.1* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 52.206.238.7:22 as 'root'
debug3: hostkeys_foreach: reading file "/Users/pivotal/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/pivotal/.ssh/known_hosts:52
debug3: load_hostkeys: loaded 1 keys from 52.206.238.7
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-ed25519,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: kex_parse_kexinit: aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: kex_parse_kexinit: aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: kex_parse_kexinit: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,hmac-sha1
debug2: kex_parse_kexinit: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,hmac-sha1
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug1: kex: server->client aes128-ctr umac-128-etm@openssh.com none
debug1: kex: client->server aes128-ctr umac-128-etm@openssh.com none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: [REDACTED]
debug3: hostkeys_foreach: reading file "/Users/pivotal/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/pivotal/.ssh/known_hosts:52
debug3: load_hostkeys: loaded 1 keys from 52.206.238.7
debug1: Host '52.206.238.7' is known and matches the ECDSA host key.
debug1: Found key in /Users/pivotal/.ssh/known_hosts:52
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /Volumes/jaresty/.ssh/work_id_rsa (0x7f916c801430),
debug2: key: /tmp/hagrid-key (0x7f916bc22660), explicit
debug3: input_userauth_banner
Unauthorized use is strictly prohibited. All access and activity
is subject to logging and monitoring.
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Volumes/jaresty/.ssh/work_id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug1: Offering RSA public key: /tmp/hagrid-key
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).

Here is a log of our cloudformation events:


2016-08-17  Status  Type    Logical ID  Status reason
16:20:44 UTC-0700   UPDATE_COMPLETE AWS::CloudFormation::Stack  bbl-aws-KVJis   
16:20:43 UTC-0700   UPDATE_COMPLETE_CLEANUP_IN_PROGRESS AWS::CloudFormation::Stack  bbl-aws-KVJis   
16:20:38 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet4RouteTableAssociation    
16:20:37 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup CFRouterInternalSecurityGroup   
16:20:36 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFRouterInternalSecurityGroup   Resource creation Initiated
16:20:35 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup CFSSHProxyInternalSecurityGroup 
16:20:34 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet3RouteTableAssociation    
16:20:34 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet1RouteTableAssociation    
16:20:33 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFSSHProxyInternalSecurityGroup Resource creation Initiated
16:20:32 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet2RouteTableAssociation    
16:20:23 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet4RouteTableAssociation    Resource creation Initiated
16:20:22 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet4RouteTableAssociation    
16:20:20 UTC-0700   CREATE_COMPLETE AWS::ElasticLoadBalancing::LoadBalancer CFRouterLoadBalancer    
16:20:20 UTC-0700   CREATE_COMPLETE AWS::ElasticLoadBalancing::LoadBalancer CFSSHProxyLoadBalancer  
16:20:20 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFRouterInternalSecurityGroup   
16:20:19 UTC-0700   CREATE_IN_PROGRESS  AWS::ElasticLoadBalancing::LoadBalancer CFSSHProxyLoadBalancer  Resource creation Initiated
16:20:19 UTC-0700   CREATE_IN_PROGRESS  AWS::ElasticLoadBalancing::LoadBalancer CFRouterLoadBalancer    Resource creation Initiated
16:20:19 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet3RouteTableAssociation    Resource creation Initiated
16:20:18 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet1RouteTableAssociation    Resource creation Initiated
16:20:18 UTC-0700   CREATE_IN_PROGRESS  AWS::ElasticLoadBalancing::LoadBalancer CFRouterLoadBalancer    
16:20:18 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet3RouteTableAssociation    
16:20:18 UTC-0700   CREATE_IN_PROGRESS  AWS::ElasticLoadBalancing::LoadBalancer CFSSHProxyLoadBalancer  
16:20:17 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet1RouteTableAssociation    
16:20:17 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFSSHProxyInternalSecurityGroup 
16:20:17 UTC-0700   CREATE_COMPLETE AWS::EC2::Route LoadBalancerRoute   
16:20:16 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet2RouteTableAssociation    Resource creation Initiated
16:20:15 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   LoadBalancerSubnet2RouteTableAssociation    
16:20:13 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup CFRouterSecurityGroup   
16:20:13 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    LoadBalancerSubnet1 
16:20:12 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup CFSSHProxySecurityGroup 
16:20:12 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    LoadBalancerSubnet4 
16:20:12 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    LoadBalancerSubnet3 
16:20:12 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    LoadBalancerSubnet2 
16:20:11 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFSSHProxySecurityGroup Resource creation Initiated
16:20:11 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFRouterSecurityGroup   Resource creation Initiated
16:20:01 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route LoadBalancerRoute   Resource creation Initiated
16:19:59 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route LoadBalancerRoute   
16:19:56 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet1 Resource creation Initiated
16:19:56 UTC-0700   CREATE_COMPLETE AWS::EC2::RouteTable    LoadBalancerRouteTable  
16:19:56 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet4 Resource creation Initiated
16:19:56 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet1 
16:19:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFSSHProxySecurityGroup 
16:19:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet3 Resource creation Initiated
16:19:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet2 Resource creation Initiated
16:19:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet4 
16:19:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    LoadBalancerRouteTable  Resource creation Initiated
16:19:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet3 
16:19:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    LoadBalancerSubnet2 
16:19:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup CFRouterSecurityGroup   
16:19:55 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    LoadBalancerRouteTable  
16:19:49 UTC-0700   UPDATE_IN_PROGRESS  AWS::CloudFormation::Stack  bbl-aws-KVJis   User Initiated
16:09:23 UTC-0700   CREATE_COMPLETE AWS::CloudFormation::Stack  bbl-aws-KVJis   
16:09:20 UTC-0700   CREATE_COMPLETE AWS::EC2::EIP   NATEIP  
16:09:04 UTC-0700   CREATE_COMPLETE AWS::EC2::Route InternalRoute   
16:08:48 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route InternalRoute   Resource creation Initiated
16:08:48 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::EIP   NATEIP  Resource creation Initiated
16:08:47 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route InternalRoute   
16:08:47 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::EIP   NATEIP  
16:08:44 UTC-0700   CREATE_COMPLETE AWS::EC2::Instance  NATInstance 
16:07:56 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Instance  NATInstance Resource creation Initiated
16:07:55 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromBOSH 
16:07:55 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromBOSH 
16:07:54 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Instance  NATInstance 
16:07:54 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromBOSH Resource creation Initiated
16:07:54 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromBOSH Resource creation Initiated
16:07:54 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromBOSH 
16:07:53 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromBOSH 
16:07:50 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup BOSHSecurityGroup   
16:07:50 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup NATSecurityGroup    
16:07:48 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup BOSHSecurityGroup   Resource creation Initiated
16:07:48 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup NATSecurityGroup    Resource creation Initiated
16:07:46 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet2RouteTableAssociation    
16:07:46 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet4RouteTableAssociation    
16:07:46 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet3RouteTableAssociation    
16:07:45 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   BOSHSubnetRouteTableAssociation 
16:07:45 UTC-0700   CREATE_COMPLETE AWS::EC2::SubnetRouteTableAssociation   InternalSubnet1RouteTableAssociation    
16:07:44 UTC-0700   CREATE_COMPLETE AWS::EC2::Route BOSHRoute   
16:07:44 UTC-0700   CREATE_COMPLETE AWS::EC2::EIP   BOSHEIP 
16:07:34 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromSelf 
16:07:33 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromSelf 
16:07:33 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromSelf Resource creation Initiated
16:07:32 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressUDPfromSelf 
16:07:32 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup BOSHSecurityGroup   
16:07:32 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromSelf Resource creation Initiated
16:07:32 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroupIngress  InternalSecurityGroupIngressTCPfromSelf 
16:07:32 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup NATSecurityGroup    
16:07:30 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet3RouteTableAssociation    Resource creation Initiated
16:07:30 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet2RouteTableAssociation    Resource creation Initiated
16:07:30 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet4RouteTableAssociation    Resource creation Initiated
16:07:29 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet3RouteTableAssociation    
16:07:29 UTC-0700   CREATE_COMPLETE AWS::IAM::AccessKey BOSHUserAccessKey   
16:07:29 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   BOSHSubnetRouteTableAssociation Resource creation Initiated
16:07:29 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet1RouteTableAssociation    Resource creation Initiated
16:07:29 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet2RouteTableAssociation    
16:07:29 UTC-0700   CREATE_IN_PROGRESS  AWS::IAM::AccessKey BOSHUserAccessKey   Resource creation Initiated
16:07:29 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet4RouteTableAssociation    
16:07:28 UTC-0700   CREATE_COMPLETE AWS::EC2::SecurityGroup InternalSecurityGroup   
16:07:28 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route BOSHRoute   Resource creation Initiated
16:07:28 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::EIP   BOSHEIP Resource creation Initiated
16:07:28 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   BOSHSubnetRouteTableAssociation 
16:07:28 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SubnetRouteTableAssociation   InternalSubnet1RouteTableAssociation    
16:07:28 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Route BOSHRoute   
16:07:28 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::EIP   BOSHEIP 
16:07:27 UTC-0700   CREATE_IN_PROGRESS  AWS::IAM::AccessKey BOSHUserAccessKey   
16:07:27 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup InternalSecurityGroup   Resource creation Initiated
16:07:26 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    InternalSubnet3 
16:07:26 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    InternalSubnet2 
16:07:26 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    InternalSubnet4 
16:07:25 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    InternalSubnet1 
16:07:25 UTC-0700   CREATE_COMPLETE AWS::EC2::Subnet    BOSHSubnet  
16:07:24 UTC-0700   CREATE_COMPLETE AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    
16:07:23 UTC-0700   CREATE_COMPLETE AWS::IAM::User  BOSHUser    
16:07:12 UTC-0700   CREATE_COMPLETE AWS::EC2::RouteTable    InternalRouteTable  
16:07:11 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    InternalRouteTable  Resource creation Initiated
16:07:10 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    InternalRouteTable  
16:07:10 UTC-0700   CREATE_COMPLETE AWS::EC2::RouteTable    BOSHRouteTable  
16:07:10 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet2 Resource creation Initiated
16:07:09 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet3 Resource creation Initiated
16:07:09 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet4 Resource creation Initiated
16:07:09 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet2 
16:07:09 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    BOSHRouteTable  Resource creation Initiated
16:07:09 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet1 Resource creation Initiated
16:07:09 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    BOSHSubnet  Resource creation Initiated
16:07:09 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet3 
16:07:09 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    Resource creation Initiated
16:07:08 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet4 
16:07:08 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::RouteTable    BOSHRouteTable  
16:07:08 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    InternalSubnet1 
16:07:08 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::VPCGatewayAttachment  VPCGatewayAttachment    
16:07:08 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::Subnet    BOSHSubnet  
16:07:08 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::SecurityGroup InternalSecurityGroup   
16:07:05 UTC-0700   CREATE_COMPLETE AWS::EC2::VPC   VPC 
16:07:03 UTC-0700   CREATE_COMPLETE AWS::EC2::InternetGateway   VPCGatewayInternetGateway   
16:06:48 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::VPC   VPC Resource creation Initiated
16:06:47 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::VPC   VPC 
16:06:47 UTC-0700   CREATE_IN_PROGRESS  AWS::IAM::User  BOSHUser    Resource creation Initiated
16:06:47 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::InternetGateway   VPCGatewayInternetGateway   Resource creation Initiated
16:06:46 UTC-0700   CREATE_IN_PROGRESS  AWS::IAM::User  BOSHUser    
16:06:46 UTC-0700   CREATE_IN_PROGRESS  AWS::EC2::InternetGateway   VPCGatewayInternetGateway   
16:06:42 UTC-0700   CREATE_IN_PROGRESS  AWS::CloudFormation::Stack  bbl-aws-KVJis   User Initiated

As the result of this, we are planning to destroy and re-setup our infrastructure. Let us know if you guys need anymore information.

@jaresty && Chunyi

cf-gitbot commented 7 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/128790821

The labels on this github issue will be updated when the story is started.

Amit-PivotalLabs commented 7 years ago

Try ssh'ing in as vcap rather than root, let me know how that goes.

ChunyiLyu commented 7 years ago

That solved it! Thanks Amit.