search

hashicorp / terraform-provider-dns

Utility provider that supports DNS updates (RFC 2136) and can optionally be configured with secret key based transaction authentication (RFC 2845).
https://registry.terraform.io/providers/hashicorp/dns/latest
Mozilla Public License 2.0
113 stars 72 forks source link

Windows Server DNS - The message or signature supplied for verification has been altered #215

Open storkeyp opened 2 years ago

storkeyp commented 2 years ago

Terraform CLI and Provider Versions

Terraform v1.2.1 on windows_amd64

Terraform Configuration

provider "dns" {
    update {
      server                  = "${var.domain_controller}.${var.domain_name}"
      gssapi {
        realm                 = upper("${var.domain_name}")
        username              = var.username
        password              = var.password
      }
    }
}

resource "dns_a_record_set" "wyse" {
    zone = "${var.dns_cname}.${var.dns_zone}"
    addresses = [
        phpipam_address.wms_ip_address[0].ip_address
    ]
}

Expected Behavior

DNS A record added to zone, or modified if it already exists

Actual Behavior

A record is successfully created, but Terraform apply fails with the following error:

Error: Error updating DNS record: The message or signature supplied for verification has been altered

with module.vms.dns_a_record_set.wyse, on ..\modules\vms\wms.tf line 99, in resource "dns_a_record_set" "wyse": 99: resource "dns_a_record_set" "wyse" {

Steps to Reproduce

  1. terraform apply

How much impact is this issue causing?

Medium

Logs

No response

Additional Information

Windows Server 2012 R2 Standard, domain integrated zone

Code of Conduct

SpeedHighway commented 2 years ago

This one's occuring for us, as well, but we're using a cname and doing an edit, for an imported record. Unfortunately, the edit did NOT occur, in our case.

EDIT: For clarification, we're trying to add/edit entries to 3 different DNS servers, and getting this result on all 3.

EDIT2: Is it possible that the RR messages are being created incorrectly? I find it hard to believe it wouldn't have been noticed before now if this is a problem, but all of the test cases inside the dns project's repository use the phrases "IN A" or "IN CNAME", whereas, this project is setting them up as "A" or "CNAME", without the word "IN". I don't know if it makes a difference, but wonder if it could. https://github.com/miekg/dns/blob/40060b4a4b85b14867003343f64bfd5cd64ea256/duplicate_test.go

mtlucas commented 2 years ago

I had the same problem and fixed it by changing my DNS zone Properties to "Dynamic updates: Secure only". It was failing when I had "Nonsecure and Secure" selected.

EDIT: OK, this happened again even after changing to secure updates. Seems intermittent and still problematic.

RahmanBadru commented 2 years ago

Has anybody been able to solve this, i have the same issue and hence i cant destroy the records and achieve desired state

ranExl commented 2 years ago

Same here. Did anyone manage to pass this one?

Aiikon commented 2 years ago

Try making your DNS server name all lowercase if it's not.

I had the exact same issue. I finally tested on a RHEL server and got a similar sounding error mentioned in #213 and the fix was the same. Changing server to lowercase worked for both Windows and Linux.

ranExl commented 2 years ago

Getting a different error now - Error: Error updating DNS record: read udp _:65428->_:53: i/o timeout

Aiikon commented 1 year ago

I've found this error also occurs with AD/GSSAPI on the Windows when multiple records are being updated simultaneously. It seems somewhere in the pipeline the DNS module is not thread safe. I've been able to work around the issue forcing a single thread using "terraform apply -parallelism 1" but this is only usable if DNS resources aren't paired with resources that take a long time to provision like VMs. I haven't had a chance to test if parallelism is a problem or not on Linux. I wish there was a configuration to control the thread count per-resource or provider to make this easier.

jayo86 commented 1 year ago

Try making your DNS server name all lowercase if it's not.

I had the exact same issue. I finally tested on a RHEL server and got a similar sounding error mentioned in #213 and the fix was the same. Changing server to lowercase worked for both Windows and Linux.

fyi this fixed it for me

cbus-guy commented 1 year ago

We have the same issue. Don't know why this isn't getting any traction. ???

bendbennett commented 7 months ago

Hi @storkeyp 👋

Sorry you ran into trouble here. Can I confirm the following:

If the issue persists, then it would be very helpful if you could gather the details necessary to reproduce this issue in a local development environment. Thanks.

rahmnstein commented 1 week ago

Just installed a new test domain controller. OS: 2022 Server FQDN: dc1.mylab.corp Forest / Domain: mylab.corp

Ran the code on the domain controller itself.

Four "terraform apply" below shows the error.

  1. It succeed with 2 / 4 records.
  2. It succeed with remaining 2 records.
  3. Success, can refresh all four records! "No changes. Your infrastructure matches the configuration."
  4. Fail to refresh all four records.

If I change 'server = "dc1.mylab.corp"' to server = "mylab.corp" I get error 'The message or signature supplied for verification has been altered' instead.

It is like it logs in once for every record at the same time and the DC says no. If I only create one DNS-record and then remove it again, it never fails. If I have more than one record in the state, it need to refresh all of them at the same time and it fails sometimes. Not every time but often.

Do you have a lab where you actually can add multiple DNS-records in the same apply on a AD domain controller?

terraform {
  required_providers {
    dns = {
      source  = "hashicorp/dns"
      version = "3.4.2"
    }
  }
}

provider "dns" {
  update {
    server = "dc1.mylab.corp"
    gssapi {
      realm    = "MYLAB.CORP"
      username = "administrator"
      password = "AmazingPw123!!"
    }
  }
}

resource "dns_a_record_set" "records" {
  count = 4

  zone = "mylab.corp."
  name = "testing${count.index + 1}"
  addresses = [
    "1.2.3.4",
  ]
  ttl = 300
}

C:\tf>.\terraform.exe apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the
following symbols:
  + create

Terraform will perform the following actions:

  # dns_a_record_set.records[0] will be created
  + resource "dns_a_record_set" "records" {
      + addresses = [
          + "1.2.3.4",
        ]
      + id        = (known after apply)
      + name      = "testing1"
      + ttl       = 300
      + zone      = "mylab.corp."
    }

  # dns_a_record_set.records[1] will be created
  + resource "dns_a_record_set" "records" {
      + addresses = [
          + "1.2.3.4",
        ]
      + id        = (known after apply)
      + name      = "testing2"
      + ttl       = 300
      + zone      = "mylab.corp."
    }

  # dns_a_record_set.records[2] will be created
  + resource "dns_a_record_set" "records" {
      + addresses = [
          + "1.2.3.4",
        ]
      + id        = (known after apply)
      + name      = "testing3"
      + ttl       = 300
      + zone      = "mylab.corp."
    }

  # dns_a_record_set.records[3] will be created
  + resource "dns_a_record_set" "records" {
      + addresses = [
          + "1.2.3.4",
        ]
      + id        = (known after apply)
      + name      = "testing4"
      + ttl       = 300
      + zone      = "mylab.corp."
    }

Plan: 4 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

dns_a_record_set.records[3]: Creating...
dns_a_record_set.records[0]: Creating...
dns_a_record_set.records[1]: Creating...
dns_a_record_set.records[2]: Creating...
dns_a_record_set.records[3]: Creation complete after 0s [id=testing4.mylab.corp.]
dns_a_record_set.records[2]: Creation complete after 0s [id=testing3.mylab.corp.]
╷
│ Error: Error updating DNS record: dns: no secrets defined
│
│   with dns_a_record_set.records[0],
│   on dns.tf line 1, in resource "dns_a_record_set" "records":
│    1: resource "dns_a_record_set" "records" {
│
╵
╷
│ Error: Error updating DNS record: dns: no secrets defined
│
│   with dns_a_record_set.records[1],
│   on dns.tf line 1, in resource "dns_a_record_set" "records":
│    1: resource "dns_a_record_set" "records" {
│
╵

C:\tf>.\terraform.exe apply
dns_a_record_set.records[3]: Refreshing state... [id=testing4.mylab.corp.]
dns_a_record_set.records[2]: Refreshing state... [id=testing3.mylab.corp.]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the
following symbols:
  + create

Terraform will perform the following actions:

  # dns_a_record_set.records[0] will be created
  + resource "dns_a_record_set" "records" {
      + addresses = [
          + "1.2.3.4",
        ]
      + id        = (known after apply)
      + name      = "testing1"
      + ttl       = 300
      + zone      = "mylab.corp."
    }

  # dns_a_record_set.records[1] will be created
  + resource "dns_a_record_set" "records" {
      + addresses = [
          + "1.2.3.4",
        ]
      + id        = (known after apply)
      + name      = "testing2"
      + ttl       = 300
      + zone      = "mylab.corp."
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

dns_a_record_set.records[0]: Creating...
dns_a_record_set.records[1]: Creating...
dns_a_record_set.records[0]: Creation complete after 0s [id=testing1.mylab.corp.]
dns_a_record_set.records[1]: Creation complete after 0s [id=testing2.mylab.corp.]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

C:\tf>.\terraform.exe apply
dns_a_record_set.records[2]: Refreshing state... [id=testing3.mylab.corp.]
dns_a_record_set.records[0]: Refreshing state... [id=testing1.mylab.corp.]
dns_a_record_set.records[3]: Refreshing state... [id=testing4.mylab.corp.]
dns_a_record_set.records[1]: Refreshing state... [id=testing2.mylab.corp.]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are
needed.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

C:\tf>.\terraform.exe apply
dns_a_record_set.records[1]: Refreshing state... [id=testing2.mylab.corp.]
dns_a_record_set.records[0]: Refreshing state... [id=testing1.mylab.corp.]
dns_a_record_set.records[3]: Refreshing state... [id=testing4.mylab.corp.]
dns_a_record_set.records[2]: Refreshing state... [id=testing3.mylab.corp.]

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Error querying DNS record: error negotiating GSS context: 1 error occurred:
│       * DNS error: REFUSED (5)
│
│
│
│   with dns_a_record_set.records[0],
│   on dns.tf line 1, in resource "dns_a_record_set" "records":
│    1: resource "dns_a_record_set" "records" {
│
╵
╷
│ Error: Error querying DNS record: dns: no secrets defined
│
│   with dns_a_record_set.records[3],
│   on dns.tf line 1, in resource "dns_a_record_set" "records":
│    1: resource "dns_a_record_set" "records" {
│
╵

C:\tf>.\terraform.exe version
Terraform v1.9.8
on windows_amd64
+ provider registry.terraform.io/hashicorp/dns v3.4.2
Aiikon commented 6 days ago

I see that parallel CRUDs are still an issue with the latest version of this provider. Hopefully this explanation and example helps some people or gets some traction towards fixing the issue. I have a customized provider that I use internally at work that you can use if you're willing to build your own copy of the provider so scroll down if you want to try that.

First make sure you are able to update a single record reliably. A few issues that may result in cryptic errors are:

The DNS error: REFUSED errors seem to be because some component of the DNS or kerberos library has threading issues. This can be tested by running terraform plan/apply/destroy with the --parallelism 1 option. Here is a sample configuration that will fail in a lab environment every time unless ran with terraform apply --parallelism 1 (at which point it succeeds 100% of the time):

terraform {
  required_providers {
    dns = {
      source = "hashicorp/dns"
      version = "3.4.2"
    }
  }
}

provider "dns" {
  update {
  server = "win-qbs0pliia8g"
    gssapi {
      realm = "domain1.local"
      username = "Administrator"
      password = "------------"
    }
  }
}

locals {
  records = ["a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k"]
}

resource "dns_a_record_set" records {
  for_each = toset(local.records)

  zone = "domain1.local."
  name = "server-${each.key}"
  addresses = ["10.0.0.99"]
}

If you're only managing DNS or other instantaneous API calls you may be happy running everything with the parallelism flag. It'll still be pretty fast. If you're running other operations like VM provisioning though you probably want everything else running in parallel, so until the provider is fixed you can try my workaround.

Adding a few mutex lock/unlock lines to the resource files and building your own provider is really easy. It's a hack, probably the wrong way to do it, and messy but I use an altered provider in a large VM build configuration and it works for both windows and linux clients (and TFE if you publish it internally). Terraform isn't aware only one resource can be accessed at a time so the logs may show some updates taking longer than they really do.

To customize the provider so only one CRUD operation can be performed at a time:

provider.go needs a global variable added near the top:

var mutex sync.Mutex

and each resource_*.go file needs a mutex lock and unlock added at the start of each CRUD function:

func (d *dnsCNAMERecordResource) Read(ctx context.Context, req resource.ReadRequest, resp *resource.ReadResponse) {

mutex.Lock()
defer mutex.Unlock()

The A record's read function is also used by update so it needs another definition that doesn't have a mutex or the provider will wait forever, it's harder to explain so check the fork I linked.

Here's a fork containing updates to the A and CNAME resource files. I've only tested these resources so you would need to add the mutex to other resources like AAAA or SRV yourself :

Building the provider is the easy part. Just install go and build with go build and it should complete nearly instantly (at least it does for me with vscode). Getting terraform to use it can be trickier. Some of your options are:

I don't want this to become a troubleshooting thread for using in-house providers so I'll leave it at that.