search

equinix / terraform-provider-equinix

Terraform Equinix provider
https://deploy.equinix.com/labs/terraform-provider-equinix/
MIT License
47 stars 45 forks source link

metal_spot_market_request - impossible to create right now #164

Open colemickens opened 2 years ago

colemickens commented 2 years ago

The log below shows:

and then an error message about capacity in "ty11".

Am I doing something wrong?

+ /nix/store/zrbhhj9aw9x945a1c0j9bq4dzg8zz9c0-terraform-1.0.9/bin/terraform -chdir=./cloud/_tf/_state apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # metal_spot_market_request.bldr-a64 will be created
  + resource "metal_spot_market_request" "bldr-a64" {
      + devices_max      = 1
      + devices_min      = 1
      + facilities       = [
          + "sjc1",
        ]
      + id               = (known after apply)
      + max_bid_price    = 0.5
      + project_id       = "afc67974-ff22-41fd-9346-5b2c8d51e3a9"
      + wait_for_devices = true

      + instance_parameters {
          + always_pxe        = false
          + billing_cycle     = "hourly"
          + hostname          = "bldr-a64"
          + operating_system  = "ubuntu_18_04"
          + plan              = "c2.large.arm"
          + termintation_time = (known after apply)
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

metal_spot_market_request.bldr-a64: Creating...
╷
│ Error: POST https://api.equinix.com/metal/v1/projects/afc67974-ff22-41fd-9346-5b2c8d51e3a9/spot-market-requests?include=devices%2Cproject%2Cplan: 422 No available servers with plan c2.large.arm in facility ty11 
│ 
│   with metal_spot_market_request.bldr-a64,
│   on config.tf.json line 1, in resource.metal_spot_market_request.bldr-a64:
│    1: {"provider":{"metal":[null]},"resource":{"metal_spot_market_request":{"bldr-a64":{"devices_max":1,"devices_min":1,"facilities":["sjc1"],"instance_parameters":{"billing_cycle":"hourly","hostname":"bldr-a64","operating_system":"ubuntu_18_04","plan":"c2.large.arm"},"max_bid_price":"0.5","project_id":"afc67974-ff22-41fd-9346-5b2c8d51e3a9","wait_for_devices":true}}},"terraform":{"required_providers":{"metal":{"source":"equinix/metal","version":"3.2.0"}}}}
│ 
╵
+ v
+ /nix/store/zrbhhj9aw9x945a1c0j9bq4dzg8zz9c0-terraform-1.0.9/bin/terraform -chdir=./cloud/_tf/_state version
Terraform v1.0.9
on linux_amd64
+ provider registry.terraform.io/equinix/metal v3.2.0

note:

colemickens commented 2 years ago

The facility I requested DOES have that plan available at that bid price, too, I just checked.

t0mk commented 2 years ago

Hey @colemickens thanks for reporting this. You have sjc1 in your configuration and the API is replying about capacity in ty11, that's very strange. I can't see right now where the problem is, so we will need to place it eitehr to the provider logic, or to the API.

Can you please run TF_LOG=DEBUG and TF_LOG_PATH=log.txt terraform apply ? That will raise verbosity of the provider, and print HTTP traffic and quite a lot of debug messages to log.txt. You can then upload log.txt here (as an attachment, please don't paste it), and I will take a look which faciltiy is in the POST request creating the spot market request (You can take a look yourself actually, if you care).

IMPORTANT - before you upload the log, please remove lines with X-Auth-Token, that's HTTP header containing you API token.

colemickens commented 2 years ago

I snippet a bit off the top, I think this is what you need:

2021-10-19T15:33:55.453-0700 [DEBUG] Starting graph walk: walkApply
2021-10-19T15:33:55.454-0700 [DEBUG] created provider logger: level=debug
2021-10-19T15:33:55.454-0700 [INFO]  provider: configuring client automatic mTLS
2021-10-19T15:33:55.473-0700 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/equinix/metal/3.2.0/linux_amd64/terraform-provider-metal_v3.2.0 args=[.terraform/providers/registry.terraform.io/equinix/metal/3.2.0/linux_amd64/terraform-provider-metal_v3.2.0]
2021-10-19T15:33:55.474-0700 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/equinix/metal/3.2.0/linux_amd64/terraform-provider-metal_v3.2.0 pid=1933169
2021-10-19T15:33:55.474-0700 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/equinix/metal/3.2.0/linux_amd64/terraform-provider-metal_v3.2.0
2021-10-19T15:33:55.480-0700 [INFO]  provider.terraform-provider-metal_v3.2.0: configuring server automatic mTLS: timestamp=2021-10-19T15:33:55.480-0700
2021-10-19T15:33:55.502-0700 [DEBUG] provider.terraform-provider-metal_v3.2.0: plugin address: address=/tmp/plugin103660321 network=unix timestamp=2021-10-19T15:33:55.502-0700
2021-10-19T15:33:55.502-0700 [DEBUG] provider: using plugin: version=5
2021-10-19T15:33:55.531-0700 [WARN]  ValidateProviderConfig from "provider[\"registry.terraform.io/equinix/metal\"]" changed the config value, but that value is unused
2021-10-19T15:33:55.533-0700 [WARN]  Provider "registry.terraform.io/equinix/metal" produced an invalid plan for metal_spot_market_request.bldr-a64, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .instance_parameters[0].always_pxe: planned value cty.False for a non-computed attribute
2021-10-19T15:33:55.533-0700 [INFO]  Starting apply for metal_spot_market_request.bldr-a64
2021-10-19T15:33:55.534-0700 [DEBUG] metal_spot_market_request.bldr-a64: applying the planned Create change
2021-10-19T15:33:55.534-0700 [DEBUG] provider.terraform-provider-metal_v3.2.0: 2021/10/19 15:33:55 [DEBUG] POST https://api.equinix.com/metal/v1/projects/afc67974-ff22-41fd-9346-5b2c8d51e3a9/spot-market-requests?include=devices%2Cproject%2Cplan
2021-10-19T15:33:55.534-0700 [INFO]  provider.terraform-provider-metal_v3.2.0: 2021/10/19 15:33:55 [DEBUG] Equinix Metal API Request Details:
---[ REQUEST ]---------------------------------------
POST /metal/v1/projects/afc67974-ff22-41fd-9346-5b2c8d51e3a9/spot-market-requests?include=devices%2Cproject%2Cplan HTTP/1.1
Host: api.equinix.com
User-Agent: HashiCorp Terraform/0.11+compatible (+https://www.terraform.io) Terraform Plugin SDK/2.7.0 terraform-provider-metal/dev packngo/v0.19.1
Connection: close
Content-Length: 229
Accept: application/json
Content-Type: application/json
X-Auth-Token: METAL_AUTH_TOKEN_REDACTED
X-Consumer-Token: aZ9GmqHTPtxevvFq9SK3Pi2yr9YCbRzduCSXF2SNem5sjB91mDq7Th3ZwTtRqMWZ
Accept-Encoding: gzip

{
 "devices_max": 1,
 "devices_min": 1,
 "facilities": [
  "sjc1"
 ],
 "max_bid_price": 0.5,
 "instance_parameters": {
  "billing_cycle": "hourly",
  "hostname": "bldr-a64",
  "operating_system": "ubuntu_18_04",
  "plan": "c2.large.arm",
  "tags": null,
  "userdata": ""
 }
}

-----------------------------------------------------: timestamp=2021-10-19T15:33:55.534-0700
2021-10-19T15:33:55.932-0700 [INFO]  provider.terraform-provider-metal_v3.2.0: 2021/10/19 15:33:55 [DEBUG] Equinix Metal API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 422 Unprocessable Entity
Connection: close
Content-Length: 75
Cache-Control: no-cache
Content-Type: application/json; charset=utf-8
Date: Tue, 19 Oct 2021 22:33:55 GMT
Strict-Transport-Security: max-age=15724800; includeSubDomains
X-Request-Id: 774fbcdefebf4302d8ed64b028c8d910

{
 "errors": [
  "No available servers with plan c2.large.arm in facility ty11"
 ]
}
-----------------------------------------------------: timestamp=2021-10-19T15:33:55.932-0700
2021-10-19T15:33:55.936-0700 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-10-19T15:33:55.937-0700 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/equinix/metal/3.2.0/linux_amd64/terraform-provider-metal_v3.2.0 pid=1933169
2021-10-19T15:33:55.937-0700 [DEBUG] provider: plugin exited
2021-10-19T15:33:55.960-0700 [DEBUG] Adding temp file log sink: /tmp/terraform-log100885863
2021-10-19T15:33:55.960-0700 [INFO]  Terraform version: 1.0.9
2021-10-19T15:33:55.960-0700 [INFO]  Go runtime version: go1.16.9
2021-10-19T15:33:55.960-0700 [INFO]  CLI args: []string{"/nix/store/vc3fc3rjxv6hypc8g7kdsaigzyswy7g5-terraform-1.0.9/bin/terraform", "-chdir=./cloud/_tf/_state", "version"}
2021-10-19T15:33:55.960-0700 [DEBUG] Attempting to open CLI config file: /home/cole/.terraformrc
2021-10-19T15:33:55.960-0700 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2021-10-19T15:33:55.960-0700 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2021-10-19T15:33:55.960-0700 [DEBUG] ignoring non-existing provider search directory /home/cole/.terraform.d/plugins
2021-10-19T15:33:55.961-0700 [DEBUG] ignoring non-existing provider search directory /home/cole/.local/share/terraform/plugins
2021-10-19T15:33:55.961-0700 [DEBUG] ignoring non-existing provider search directory /nix/store/wmbkcc32p0jggfrk1fgivp80gw2isfys-nix-2.5pre20211007_844dd90/share/terraform/plugins
2021-10-19T15:33:55.961-0700 [DEBUG] ignoring non-existing provider search directory /nix/store/b3cnyv7j58b9241w6scdx0mxn4rqw2np-cachix-0.6.1/share/terraform/plugins
2021-10-19T15:33:55.961-0700 [DEBUG] ignoring non-existing provider search directory /nix/store/l5q87hfka3ja0kzlw4ja4kffcimhiasf-mercurial-5.9.1/share/terraform/plugins
2021-10-19T15:33:55.961-0700 [DEBUG] ignoring non-existing provider search directory /nix/store/97w52ckcjnfiz89h3lh7zf1kysgfm2s8-python3-3.9.6/share/terraform/plugins
2021-10-19T15:33:55.961-0700 [DEBUG] ignoring non-existing provider search directory /nix/store/32r66ar7hxcc4a6hh26yx49w79bjhnw7-git-2.33.0/share/terraform/plugins
2021-10-19T15:33:55.961-0700 [DEBUG] ignoring non-existing provider search directory /nix/store/dvpq8d2h53p6m42fcx1v6kpynzn32lr6-net-tools-2.10/share/terraform/plugins
2021-10-19T15:33:55.961-0700 [DEBUG] ignoring non-existing provider search directory /nix/store/qlhf7i79i8gqr3jm8ibf317q3v1l7wrb-openssh-8.7p1/share/terraform/plugins
2021-10-19T15:33:55.961-0700 [DEBUG] ignoring non-existing provider search directory /nix/store/f6ciksmivka1idkrn9s2vx025z8gxydh-ripgrep-13.0.0/share/terraform/plugins
2021-10-19T15:33:55.961-0700 [DEBUG] ignoring non-existing provider search directory /nix/store/g80wzbdkacw7aqgpygvndklaszrh569l-rsync-3.2.3/share/terraform/plugins
2021-10-19T15:33:55.961-0700 [DEBUG] ignoring non-existing provider search directory /nix/store/iqjf5y9qamdlgdl8955mhsrsr907rb13-gh-2.1.0/share/terraform/plugins
2021-10-19T15:33:55.961-0700 [DEBUG] ignoring non-existing provider search directory /nix/store/31pkw5yi08fj4l0glzvpf1cp4ywkxh86-gawk-5.1.0/share/terraform/plugins
2021-10-19T15:33:55.961-0700 [DEBUG] ignoring non-existing provider search directory /nix/store/dy4ylp9439la4lq35ah2mj80fi87pk4w-gnused-4.8/share/terraform/plugins
2021-10-19T15:33:55.961-0700 [DEBUG] ignoring non-existing provider search directory /nix/store/xxgddhdi57bbgd1yxza44plq6krjmiz1-gnugrep-3.6/share/terraform/plugins
2021-10-19T15:33:55.961-0700 [DEBUG] ignoring non-existing provider search directory /nix/store/s9vg3kkfbr1pqm0gnqdl72mx6viz2q6h-desktops/share/terraform/plugins
2021-10-19T15:33:55.961-0700 [DEBUG] ignoring non-existing provider search directory /home/cole/.nix-profile/share/terraform/plugins
2021-10-19T15:33:55.961-0700 [DEBUG] ignoring non-existing provider search directory /etc/profiles/per-user/cole/share/terraform/plugins
2021-10-19T15:33:55.961-0700 [DEBUG] ignoring non-existing provider search directory /nix/var/nix/profiles/default/share/terraform/plugins
2021-10-19T15:33:55.961-0700 [DEBUG] ignoring non-existing provider search directory /run/current-system/sw/share/terraform/plugins
2021-10-19T15:33:55.961-0700 [INFO]  CLI command args: []string{"version"}
t0mk commented 2 years ago

@colemickens thanks for the debug log. It's an API issue. Can you please reach to Metal support, giving them the dump of the POST request and response?

Via slack, or some other channel that you can use as a customer. Here we mostly deal with the provider functionality and I myself have no priviliges to escalate API bugs.

displague commented 2 years ago

I'm noting that this is being tracked internally in terms of unexpected API behavior.

colemickens commented 2 years ago

Just for transparency, I'm not going to follow up with Support since it seems like it got escalated internally. Thanks all for the fast turnaround and update.

For anyone else - passing sv as the metro parameter caused me to wind up provisioning in the datacenter that I wanted anyway. Could be an alternative if you don't need exact facility placement.

colemickens commented 2 years ago

So. I'm actually now blocked on this, because the API no longer seems to allow me to pass metro.

:/ this seems rather broken.

│ Error: POST https://api.equinix.com/metal/v1/projects/afc67974-ff22-41fd-9346-5b2c8d51e3a9/spot-market-requests?include=devices%2Cproject%2Cplan: 422 You are not allowed to provision c3.large.arm servers, You must specify a valid facility parameter

and if I switch back to facility, it still 500s:

╷
│ Error: POST https://api.equinix.com/metal/v1/projects/afc67974-ff22-41fd-9346-5b2c8d51e3a9/spot-market-requests?include=devices%2Cproject%2Cplan: 500 Oh snap, something went wrong! We've logged the error and will take a look - please reach out to us if you continue having trouble.
│
│   with metal_spot_market_request.pktspotarm0,
│   on config.tf.json line 25, in resource.metal_spot_market_request.pktspotarm0:
│   25:       }
│
╵

Updating the title to reflect this.

displague commented 2 years ago

@colemickens The 500 indicates that something wasn't handled properly. To narrow down the problem, could you try to recreate this SpotMarketRequest using the portal at https://console.equinix.com.

One suspicion is that there may be no availability of that plan within the metro or facility. Try checking the availability of the plan within the desired metro according to metal capacity check (https://github.com/equinix/metal-cli#installation).

Spot market device availability should be greater than on-demand availability with a sufficient bid, if any devices matching the desired plans are available in the metro).

jdmarble commented 2 years ago

I'm blocked on this too. Doesn't seem to matter what facility I specify, the response is always about hk2: 422 No available servers with plan m3.small.x86 in facility hk2 . When I use "metro", I don't get a 500 but at least the facility in the rejection message is in the requested metro.

resource "metal_spot_market_request" "spot" {
  project_id       = data.metal_project. ... .project_id
  max_bid_price    = 0.11
  facilities       = ["da11"]
  # Also fails on all of the following facilities "da11", "dc13", "se4", "sg4", "sv15", "ty11" 
  # Also fails when metro is one of "da", "dc", "sg", "sv"
  devices_min      = 1
  devices_max      = 1

  instance_parameters {
    hostname         = ...
    plan             = "m3.small.x86"
    billing_cycle    = "hourly"
    operating_system = "custom_ipxe"
    ipxe_script_url  = ...
    always_pxe       = true
  }
}

Notably, by the time I got to testing metro = "sv", all further requests were getting "403 Access denied for the current authentication token" response. Perhaps I tripped some API rate limiter? Anyway, all of my terraform apply resulted in failure, but when I checked the web console I had a spot instance running in SV!