Closed mrdavidlaing closed 11 years ago
drnic
commented
11 years ago Thanks @mrdavidlaing.
If you're able to help debugging before I can get to look at this - is there any mention of us-east-1:
bosh-bootstrap ssh), in /var/vcap/store/microboshes/deployments/microbosh-aws-eu-west-1/micro_bosh.yml
mrdavidlaing
commented
11 years ago @drnic - Thanks for the tips!
I deleted everything and started again with bundle exec bosh-bootstrap deploy --edge-stemcell --edge-deployer just to be sure this isn't an "old" bug.
Same problem occured.
So:
grep us-east-1 ~/.bosh_bootstrap/manifest.yml returns nothing.grep us-east-1 /var/vcap/store/microboshes/deployments/microbosh-aws-eu-west-1/micro_bosh.yml returns nothing.~$ sudo grep -r us-east-1 /var/vcap gives/var/vcap/store/releases/bosh-release/src/bosh/aws_cpi/lib/cloud/aws/cloud.rb: DEFAULT_AVAILABILITY_ZONE = "us-east-1a"
/var/vcap/store/releases/bosh-release/src/bosh/aws_cpi/README.md: EC2 region, defaults to `us-east-1`
/var/vcap/store/releases/bosh-release/src/bosh/aws_cpi/spec/unit/create_disk_spec.rb: :availability_zone => "us-east-1a"
/var/vcap/store/releases/bosh-release/src/bosh/aws_cpi/spec/unit/create_disk_spec.rb: :availability_zone => "us-east-1a"
/var/vcap/store/releases/bosh-release/src/bosh/aws/README: us-east-1:
/var/vcap/store/releases/bosh-release/src/bosh/aws/bin/ami: us-east-1
/var/vcap/store/releases/bosh-release/src/bosh/aws_bootstrap/lib/bosh/cli/commands/aws.rb: say("no region specified, defaulting to us-east-1")
/var/vcap/store/releases/bosh-release/src/bosh/aws_bootstrap/spec/functional/aws_spec.rb: {"cidr" => "10.10.0.0/24", "availability_zone" => "us-east-1a"},
/var/vcap/store/releases/bosh-release/src/bosh/aws_bootstrap/spec/functional/aws_spec.rb: {"cidr" => "10.10.1.0/24", "availability_zone" => "us-east-1b"}
/var/vcap/store/releases/bosh-release/src/bosh/aws_bootstrap/spec/assets/config.yml: region: us-east-1
/var/vcap/store/releases/bosh-release/src/bosh/aws_bootstrap/spec/assets/config.yml: availability_zone: us-east-1a
/var/vcap/store/releases/bosh-release/src/bosh/aws_bootstrap/spec/assets/config.yml: availability_zone: us-east-1b
/var/vcap/store/releases/bosh-release/src/bosh/aws_bootstrap/spec/assets/test-output.yml: region: us-east-1
/var/vcap/store/releases/bosh-release/src/bosh/aws_bootstrap/spec/unit/aws_spec.rb: subnets.should_receive(:create).with("1.0.10.0/24", {:availability_zone => "us-east-1"})
/var/vcap/store/releases/bosh-release/src/bosh/aws_bootstrap/spec/unit/aws_spec.rb: aws.create_subnets([{"cidr" => "1.0.10.0/24", "availability_zone" => "us-east-1"}])
/var/vcap/store/repos/bosh/aws_cpi/lib/cloud/aws/cloud.rb: DEFAULT_AVAILABILITY_ZONE = "us-east-1a"
/var/vcap/store/repos/bosh/aws_cpi/README.md: EC2 region, defaults to `us-east-1`
/var/vcap/store/repos/bosh/aws_cpi/spec/unit/create_disk_spec.rb: :availability_zone => "us-east-1a"
/var/vcap/store/repos/bosh/aws_cpi/spec/unit/create_disk_spec.rb: :availability_zone => "us-east-1a"
/var/vcap/store/repos/bosh/aws/README: us-east-1:
/var/vcap/store/repos/bosh/aws/bin/ami: us-east-1
/var/vcap/store/repos/bosh/aws_bootstrap/lib/bosh/cli/commands/aws.rb: say("no region specified, defaulting to us-east-1")
/var/vcap/store/repos/bosh/aws_bootstrap/spec/functional/aws_spec.rb: {"cidr" => "10.10.0.0/24", "availability_zone" => "us-east-1a"},
/var/vcap/store/repos/bosh/aws_bootstrap/spec/functional/aws_spec.rb: {"cidr" => "10.10.1.0/24", "availability_zone" => "us-east-1b"}
/var/vcap/store/repos/bosh/aws_bootstrap/spec/assets/config.yml: region: us-east-1
/var/vcap/store/repos/bosh/aws_bootstrap/spec/assets/config.yml: availability_zone: us-east-1a
/var/vcap/store/repos/bosh/aws_bootstrap/spec/assets/config.yml: availability_zone: us-east-1b
/var/vcap/store/repos/bosh/aws_bootstrap/spec/assets/test-output.yml: region: us-east-1
/var/vcap/store/repos/bosh/aws_bootstrap/spec/unit/aws_spec.rb: subnets.should_receive(:create).with("1.0.10.0/24", {:availability_zone => "us-east-1"})
/var/vcap/store/repos/bosh/aws_bootstrap/spec/unit/aws_spec.rb: aws.create_subnets([{"cidr" => "1.0.10.0/24", "availability_zone" => "us-east-1"}])
/var/vcap/store/microboshes/deployments/microbosh-aws-eu-west-1/bosh_micro_deploy.log:I, [2013-01-24T18:45:22.893970 #29343] [create_vm(bm-a811ea5f-ecda-462d-b88c-601484bfab0d, ...)] INFO -- : [AWS EC2 400 0.139043 0 retries] run_instances(:client_token=>"358ef71c-6656-11e2-91b4-12313c0631bf",:image_id=>"ami-d01917a4",:instance_type=>"m1.medium",:key_name=>"microbosh-aws-eu-west-1",:max_count=>1,:min_count=>1,:placement=>{:availability_zone=>"us-east-1a"},:security_groups=>["microbosh-aws-eu-west-1"],:user_data=>"eyJyZWdpc3RyeSI6eyJlbmRwb2ludCI6Imh0dHA6Ly9hZG1pbjphZG1pbkBs\nb2NhbGhvc3Q6MjU4ODgifX0=") AWS::EC2::Errors::InvalidParameterValue Invalid availability zone: [us-east-1a]Could /var/vcap/store/releases/bosh-release/src/bosh/aws_cpi/lib/cloud/aws/cloud.rb: DEFAULT_AVAILABILITY_ZONE = "us-east-1a" be a clue?
drnic
commented
11 years ago Yeah, can you run the greps again for eu-west? Perhaps I'm not populating the region at all and its defaulting to us-east-1?
drnic
commented
11 years ago You shouldn't need edge stemcells any more. Martin just release new public stemcell which map to what I've been working against. From now on "--latest-stemcell" is the flag to use (since the 0.8.1 stemcell is not marked "stable" yet).
mrdavidlaing
commented
11 years ago can you run the greps again for eu-west
I think you mean this:
$ grep eu-west ~/.bosh_bootstrap/manifest.yml
region: eu-west-1
ec2_endpoint: ec2.eu-west-1.amazonaws.com
- microbosh-aws-eu-west-1
default_key_name: microbosh-aws-eu-west-1
ec2_private_key: /home/vcap/.ssh/microbosh-aws-eu-west-1.pem
region_code: eu-west-1
bosh_name: microbosh-aws-eu-west-1
name: microbosh-aws-eu-west-1
name: microbosh-aws-eu-west-1
host: ec2-46-51-171-219.eu-west-1.compute.amazonaws.com~$ grep eu-west /var/vcap/store/microboshes/deployments/microbosh-aws-eu-west-1/micro_bosh.yml
name: microbosh-aws-eu-west-1
ec2_endpoint: ec2.eu-west-1.amazonaws.com
- microbosh-aws-eu-west-1
default_key_name: microbosh-aws-eu-west-1
ec2_private_key: /home/vcap/.ssh/microbosh-aws-eu-west-1.pem~$ sudo grep -r eu-west /var/vcap
/var/vcap/store/releases/bosh-release/.git/logs/HEAD:0000000000000000000000000000000000000000 e960acd9d6529b2500c5f98254454891c64df692 root <root@ip-10-51-50-77.eu-west-1.compute.internal> 1359048945 +0000 clone: from https://github.com/cloudfoundry/bosh-release.git
/var/vcap/store/releases/bosh-release/.git/logs/refs/heads/master:0000000000000000000000000000000000000000 e960acd9d6529b2500c5f98254454891c64df692 root <root@ip-10-51-50-77.eu-west-1.compute.internal> 1359048945 +0000 clone: from https://github.com/cloudfoundry/bosh-release.git
/var/vcap/store/releases/bosh-release/src/bosh/.git/logs/HEAD:0000000000000000000000000000000000000000 9719202c9d3fed5e77205adc156b109a9cf683f4 root <root@ip-10-51-50-77.eu-west-1.compute.internal> 1359048976 +0000 clone: from https://github.com/cloudfoundry/bosh.git
/var/vcap/store/releases/bosh-release/src/bosh/.git/logs/HEAD:9719202c9d3fed5e77205adc156b109a9cf683f4 57b31b3f15804d8df6c3ffded6a1555b5bc065f5 root <root@ip-10-51-50-77.eu-west-1.compute.internal> 1359048979 +0000 checkout: moving from master to 57b31b3f15804d8df6c3ffded6a1555b5bc065f5
/var/vcap/store/releases/bosh-release/src/bosh/.git/logs/refs/heads/master:0000000000000000000000000000000000000000 9719202c9d3fed5e77205adc156b109a9cf683f4 root <root@ip-10-51-50-77.eu-west-1.compute.internal> 1359048976 +0000 clone: from https://github.com/cloudfoundry/bosh.git
/var/vcap/store/releases/bosh-release/src/bosh/aws/bin/ami: eu-west-1
/var/vcap/store/releases/bosh-release/src/gonatsd/.git/logs/HEAD:0000000000000000000000000000000000000000 8b836c98520c9913f85aaf4419fa070f64b81668 root <root@ip-10-51-50-77.eu-west-1.compute.internal> 1359048982 +0000 clone: from https://github.com/cloudfoundry/gonatsd.git
/var/vcap/store/releases/bosh-release/src/gonatsd/.git/logs/HEAD:8b836c98520c9913f85aaf4419fa070f64b81668 8101d1446a3f208cd4cd2974d43bce7078ca39cc root <root@ip-10-51-50-77.eu-west-1.compute.internal> 1359048983 +0000 checkout: moving from master to 8101d1446a3f208cd4cd2974d43bce7078ca39cc
/var/vcap/store/releases/bosh-release/src/gonatsd/.git/logs/refs/heads/master:0000000000000000000000000000000000000000 8b836c98520c9913f85aaf4419fa070f64b81668 root <root@ip-10-51-50-77.eu-west-1.compute.internal> 1359048982 +0000 clone: from https://github.com/cloudfoundry/gonatsd.git
/var/vcap/store/repos/bosh/.git/logs/HEAD:0000000000000000000000000000000000000000 9719202c9d3fed5e77205adc156b109a9cf683f4 root <root@ip-10-51-50-77.eu-west-1.compute.internal> 1359048465 +0000 clone: from https://github.com/cloudfoundry/bosh.git
/var/vcap/store/repos/bosh/.git/logs/refs/heads/master:0000000000000000000000000000000000000000 9719202c9d3fed5e77205adc156b109a9cf683f4 root <root@ip-10-51-50-77.eu-west-1.compute.internal> 1359048465 +0000 clone: from https://github.com/cloudfoundry/bosh.git
/var/vcap/store/repos/bosh/aws/bin/ami: eu-west-1
/var/vcap/store/microboshes/deployments/bosh-deployments.yml: :name: microbosh-aws-eu-west-1
/var/vcap/store/microboshes/deployments/microbosh-aws-eu-west-1/bosh_micro_deploy.log:I, [2013-01-24T18:33:49.868528 #29343] [create_disk(4096, i-18b24b52)] INFO -- : [AWS EC2 200 0.350324 0 retries] create_volume(:availability_zone=>"eu-west-1b",:size=>4)
/var/vcap/store/microboshes/deployments/microbosh-aws-eu-west-1/bosh_micro_deploy.log:D, [2013-01-24T18:45:22.556370 #29343] [create_vm(bm-a811ea5f-ecda-462d-b88c-601484bfab0d, ...)] DEBUG -- : using security groups: microbosh-aws-eu-west-1
/var/vcap/store/microboshes/deployments/microbosh-aws-eu-west-1/bosh_micro_deploy.log:I, [2013-01-24T18:45:22.893970 #29343] [create_vm(bm-a811ea5f-ecda-462d-b88c-601484bfab0d, ...)] INFO -- : [AWS EC2 400 0.139043 0 retries] run_instances(:client_token=>"358ef71c-6656-11e2-91b4-12313c0631bf",:image_id=>"ami-d01917a4",:instance_type=>"m1.medium",:key_name=>"microbosh-aws-eu-west-1",:max_count=>1,:min_count=>1,:placement=>{:availability_zone=>"us-east-1a"},:security_groups=>["microbosh-aws-eu-west-1"],:user_data=>"eyJyZWdpc3RyeSI6eyJlbmRwb2ludCI6ImhxxxxxODgifX0=") AWS::EC2::Errors::InvalidParameterValue Invalid availability zone: [us-east-1a]
/var/vcap/store/microboshes/deployments/microbosh-aws-eu-west-1/micro_bosh.yml:name: microbosh-aws-eu-west-1
/var/vcap/store/microboshes/deployments/microbosh-aws-eu-west-1/micro_bosh.yml: ec2_endpoint: ec2.eu-west-1.amazonaws.com
/var/vcap/store/microboshes/deployments/microbosh-aws-eu-west-1/micro_bosh.yml: - microbosh-aws-eu-west-1
/var/vcap/store/microboshes/deployments/microbosh-aws-eu-west-1/micro_bosh.yml: default_key_name: microbosh-aws-eu-west-1
/var/vcap/store/microboshes/deployments/microbosh-aws-eu-west-1/micro_bosh.yml: ec2_private_key: /home/vcap/.ssh/microbosh-aws-eu-west-1.pemDoing some puts debugging in aws_cpi/lib/cloud/aws/cloud.rb
reveals that:
disk_locality: nil
resource_pool["availability_zone"]: nilMeaning that select_availability_zone(disk_locality, resource_pool["availability_zone"]) is going to fall through to default_availablity_zone()
So, taking a guess, I added availability_zone: eu-west-1b to /var/vcap/store/microboshes/deployments/microbosh-aws-eu-west-1/micro_bosh.yml
i.e
...snip...
resources:
persistent_disk: 16384
cloud_properties:
instance_type: m1.medium
availability_zone: eu-west-1b
cloud:
...snip...And viola; I advance to the next level!
Deploy Micro BOSH
unpacking stemcell (00:00:16)
uploading stemcell (00:14:39)
creating VM from ami-e2121c96 (00:00:24)
waiting for the agent (00:03:49)
...snip...In conclusion, I believe the core of the problem lies in the fact that the availability_zone: eu-west-1b is not added to /var/vcap/store/microboshes/deployments/microbosh-aws-eu-west-1/micro_bosh.yml
So, who / where is /var/vcap/store/microboshes/deployments/microbosh-aws-eu-west-1/micro_bosh.yml created, and how do we fix that ?
mrdavidlaing
commented
11 years ago Cherrypicking commit 9839edb into master has fixed this issue.
:beers: for @drnic
drnic
commented
11 years ago That's awesome. I figured that out by reading the bosh/director code on the plane.
I'll leave the ticket open until its merged into master in this repo; so I don't forget :)
bonzofenix
commented
11 years ago I just had this issue on bosh-bootstrap deploy
Deploy Micro BOSH
unpacking stemcell (00:00:20)
uploading stemcell (00:14:05)
Delete stemcell |ooo |
2/11 00:14:27 ETA: 01:04:59/usr/local/lib/ruby/gems/1.9.1/gems/aws-sdk-1.6.9/lib/aws/core/client.rb:318
:in `return_or_raise': Invalid availability zone: [us-east-1b] (AWS::EC2::Errors::InvalidParameterValue)any idea of how to solve it ?
I'm struggling to use bosh-bootstrap on AWS: eu-west-1.
When it gets to the point of deploying the Micro BOSH, is fails as below:
At a guess, I'd say that there is a config file which is hardcoded to use us-east-1a; but I'm unsure how to change that.
Full session output below (in case there is something useful in it):