search

digitalocean / terraform-provider-digitalocean

Terraform DigitalOcean provider
https://registry.terraform.io/providers/digitalocean/digitalocean/latest/docs
Mozilla Public License 2.0
503 stars 276 forks source link

DatabaseConnectionPool gives 500 error #381

Closed magsoftware closed 4 years ago

magsoftware commented 4 years ago

Terraform Version

Terraform v0.12.20

Affected Resource(s)

I create a managed database + k8s cluster. Database is ready first than the cluster. Than I want to create a firewall for database and allow traffic from k8s cluster.

Terraform Configuration Files

resource "digitalocean_database_cluster" "rdb-cluster" {
  name       = var.rdb_cluster_name
  engine     = "pg"
  version    = var.rdb_version
  size       = var.rdb_size
  node_count = var.rdb_node_count
  region     = var.project_region
}

resource "digitalocean_database_firewall" "rdb-firewall" {
  cluster_id = digitalocean_database_cluster.rdb-cluster.id

  rule {
    type  = "ip_addr"
    value = "1.2.3.4"
  }

  rule {
    type  = "k8s"
    value = digitalocean_kubernetes_cluster.flexvoucher.id
  }
}

resource "digitalocean_database_user" "rdb-user" {
  cluster_id = digitalocean_database_cluster.rdb-cluster.id
  name       = var.database_user
}

resource "digitalocean_database_db" "rdb-database" {
  cluster_id = digitalocean_database_cluster.rdb-cluster.id
  name       = var.database_name
}

resource "digitalocean_database_connection_pool" "rdb-connection-pool" {
  cluster_id = digitalocean_database_cluster.rdb-cluster.id
  name       = var.database_connection_pool_name
  mode       = var.database_connection_pool_mode
  size       = var.database_connection_pool_size
  db_name    = var.database_name
  user       = var.database_user
}

data "digitalocean_kubernetes_versions" "k8s_versions" {
  version_prefix = "1.16."
}

resource "digitalocean_kubernetes_cluster" "flexvoucher" {
  name    = var.k8s_cluster_name
  region  = var.project_region
  version = data.digitalocean_kubernetes_versions.k8s_versions.latest_version
  tags    = [var.k8s_cluster_name]

  node_pool {
    name       = var.k8s_node_pool_name
    size       = var.k8s_node_pool_size
    node_count = var.k8s_node_pool_node_count
  }
}

Debug Output

...
digitalocean_database_cluster.rdb-cluster: Still creating... [4m10s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [4m20s elapsed]
digitalocean_database_cluster.rdb-cluster: Still creating... [4m20s elapsed]
digitalocean_database_cluster.rdb-cluster: Still creating... [4m30s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [4m30s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [4m40s elapsed]
digitalocean_database_cluster.rdb-cluster: Still creating... [4m40s elapsed]
digitalocean_database_cluster.rdb-cluster: Creation complete after 4m50s [id=d1526e33-9eab-4f8b-b70c-a34f6b512dcc]
digitalocean_database_db.rdb-database: Creating...
digitalocean_database_user.rdb-user: Creating...
digitalocean_database_connection_pool.rdb-connection-pool: Creating...
digitalocean_project.flexvoucher: Creating...
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [4m50s elapsed]
digitalocean_database_user.rdb-user: Creation complete after 2s [id=d1526e33-9eab-4f8b-b70c-a34f6b512dcc/user/fv]
digitalocean_database_db.rdb-database: Creation complete after 2s [id=d1526e33-9eab-4f8b-b70c-a34f6b512dcc/database/fv]
digitalocean_project.flexvoucher: Creation complete after 3s [id=af30327c-326f-4615-a862-df131b8ddfc7]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [5m0s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [5m10s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [5m20s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [5m30s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [5m40s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [5m50s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [6m0s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [6m10s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [6m20s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Creation complete after 6m23s [id=043b82c6-9a1b-463d-94dd-8ae6a0bb3826]
digitalocean_database_firewall.rdb-firewall: Creating...
digitalocean_database_firewall.rdb-firewall: Creation complete after 2s [id=d1526e33-9eab-4f8b-b70c-a34f6b512dcc-20200219161940046400000001]

Error: Error creating DatabaseConnectionPool: POST https://api.digitalocean.com/v2/databases/d1526e33-9eab-4f8b-b70c-a34f6b512dcc/pools: 500 Server Error

  on rdb.tf line 37, in resource "digitalocean_database_connection_pool" "rdb-connection-pool":
  37: resource "digitalocean_database_connection_pool" "rdb-connection-pool" {

other

digitalocean_database_cluster.rdb-cluster: Still creating... [5m0s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [5m0s elapsed]
digitalocean_database_cluster.rdb-cluster: Still creating... [5m10s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [5m10s elapsed]
digitalocean_database_cluster.rdb-cluster: Creation complete after 5m19s [id=cae0f838-5af0-4b5b-96b4-f741e586758c]
digitalocean_database_user.rdb-user: Creating...
digitalocean_database_db.rdb-database: Creating...
digitalocean_project.flexvoucher: Creating...
digitalocean_database_connection_pool.rdb-connection-pool: Creating...
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [5m20s elapsed]
digitalocean_database_user.rdb-user: Creation complete after 2s [id=cae0f838-5af0-4b5b-96b4-f741e586758c/user/fv]
digitalocean_project.flexvoucher: Creation complete after 3s [id=5198af2e-08b0-4002-b595-38278315c955]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [5m30s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [5m40s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [5m50s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [6m0s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Creation complete after 6m4s [id=beab6c86-4ec6-4c7c-bf4c-2d0d4bbacb26]
digitalocean_database_firewall.rdb-firewall: Creating...
digitalocean_database_firewall.rdb-firewall: Creation complete after 1s [id=cae0f838-5af0-4b5b-96b4-f741e586758c-20200219163502590000000001]

Error: Provider produced inconsistent result after apply

When applying changes to digitalocean_database_db.rdb-database, provider
"registry.terraform.io/-/digitalocean" produced an unexpected new value for
was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Error: Error creating DatabaseConnectionPool: POST https://api.digitalocean.com/v2/databases/cae0f838-5af0-4b5b-96b4-f741e586758c/pools: 500 Server Error

  on rdb.tf line 37, in resource "digitalocean_database_connection_pool" "rdb-connection-pool":
  37: resource "digitalocean_database_connection_pool" "rdb-connection-pool" {

Panic Output

If Terraform produced a panic, please provide a link to a GitHub Gist containing the output of the crash.log.

Expected Behavior

This should create a database firewall rule to allow traffic from k8s cluster.

Actual Behavior

API crashes.

Steps to Reproduce

  1. terraform apply terraform.log

TF log attached.

eddiezane commented 4 years ago

@magsoftware apologies you hit this.

Was this a one time failure or is it failing every time you run it?

Could you please provide a full reproducible config that I can copy paste? i.e. with values instead of vars.

magsoftware commented 4 years ago

@eddiezane thank you for quick reply.

Was this a one time failure or is it failing every time you run it?

I get it every time I run the script.

Could you please provide a full reproducible config that I can copy paste? i.e. with values instead of vars.

All files are attached below.

files.log

magsoftware commented 4 years ago

Just a note:

same result:

digitalocean_database_cluster.rdb-cluster: Still creating... [4m50s elapsed]
digitalocean_database_cluster.rdb-cluster: Still creating... [5m0s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [5m0s elapsed]
digitalocean_database_cluster.rdb-cluster: Creation complete after 5m3s [id=8122b7d9-750b-4ea5-b160-ca403da7ecce]
digitalocean_database_user.rdb-user: Creating...
digitalocean_database_db.rdb-database: Creating...
digitalocean_database_connection_pool.rdb-connection-pool: Creating...
digitalocean_project.flexvoucher: Creating...
digitalocean_database_db.rdb-database: Creation complete after 2s [id=8122b7d9-750b-4ea5-b160-ca403da7ecce/database/fv]
digitalocean_database_user.rdb-user: Creation complete after 2s [id=8122b7d9-750b-4ea5-b160-ca403da7ecce/user/fv]
digitalocean_project.flexvoucher: Creation complete after 3s [id=4e453610-655e-4851-9dfe-f358510ea8c7]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [5m10s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [5m20s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [5m30s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [5m40s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [5m50s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [6m0s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [6m10s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [6m20s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [6m30s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [6m40s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Still creating... [6m50s elapsed]
digitalocean_kubernetes_cluster.flexvoucher: Creation complete after 6m53s [id=35e3f4f8-bca6-401a-8a30-9e71f8384790]
digitalocean_database_firewall.rdb-firewall: Creating...
digitalocean_database_firewall.rdb-firewall: Creation complete after 3s [id=8122b7d9-750b-4ea5-b160-ca403da7ecce-20200220143233376700000001]

Error: Error creating DatabaseConnectionPool: POST https://api.digitalocean.com/v2/databases/8122b7d9-750b-4ea5-b160-ca403da7ecce/pools: 500 Server Error

  on rdb.tf line 37, in resource "digitalocean_database_connection_pool" "rdb-connection-pool":
  37: resource "digitalocean_database_connection_pool" "rdb-connection-pool" {

If I run terraform apply once again:

data.digitalocean_kubernetes_versions.k8s_versions: Refreshing state...
digitalocean_spaces_bucket.flexvoucher-assets: Refreshing state... [id=flexvoucher]
digitalocean_database_cluster.rdb-cluster: Refreshing state... [id=8122b7d9-750b-4ea5-b160-ca403da7ecce]
digitalocean_kubernetes_cluster.flexvoucher: Refreshing state... [id=35e3f4f8-bca6-401a-8a30-9e71f8384790]
digitalocean_database_db.rdb-database: Refreshing state... [id=8122b7d9-750b-4ea5-b160-ca403da7ecce/database/fv]
digitalocean_database_firewall.rdb-firewall: Refreshing state... [id=8122b7d9-750b-4ea5-b160-ca403da7ecce-20200220143233376700000001]
digitalocean_database_user.rdb-user: Refreshing state... [id=8122b7d9-750b-4ea5-b160-ca403da7ecce/user/fv]
digitalocean_project.flexvoucher: Refreshing state... [id=4e453610-655e-4851-9dfe-f358510ea8c7]
digitalocean_database_connection_pool.rdb-connection-pool: Creating...
digitalocean_database_connection_pool.rdb-connection-pool: Creation complete after 3s [id=8122b7d9-750b-4ea5-b160-ca403da7ecce/fv_pool]

All good. Looks like all resources are created as specified in tf files.

Nevon commented 4 years ago

We are currently getting 500s when trying to create connection pools with names longer than 60 characters. The UI claims that the limit is between 3 and 63 characters, but creating one with >60 characters is failing.

Nevon commented 4 years ago

I found a possible explanation for the above issue. The connection pool size can be a maximum of 25 * - 3, and if you try to create a connection pool larger than that DigitalOcean's API responds with a 500 rather than any validation error. In the control panel UI they have some client-side validation checking for this, but apparently there's nothing in the API.

andrewsomething commented 4 years ago

We've added some validation around this and the DO API should now be returning a proper error message as well. I'm going to go ahead and close this one out. Please feel free to reopen if you experience this again or have additional details.