artifactory_permission_target fails to apply with error: Root object was present, but now absent.

tpham-drw commented 3 months ago

Describe the bug Terraform fails to apply artifactory_permission_target resource with error Root object was present, but now absent. on the first attempt. Second attempt is successful.

Full error message:

│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to
│ module.devx-generic-thanh-dev.artifactory_permission_target.private,
│ provider "provider[\"registry.terraform.io/jfrog/artifactory\"]" produced
│ an unexpected new value: Root object was present, but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

Requirements for and issue

[x] A description of the bug

[x] A fully functioning terraform snippet that can be copy&pasted (no outside files or ENV vars unless that's part of the issue).

resource "artifactory_local_generic_repository" "generic_repository" {
key                 = "devx-generic-thanh-dev"
description         = "Test generic local repository"
includes_pattern    = "**/*"

lifecycle {
  ignore_changes = [
    project_key,
  ]
}
}

resource "artifactory_group" "read_group" {
name             = "artifactory-repo-devx-generic-thanh-dev-read"
description      = "Test read group"
admin_privileges = false
realm            = "ldap"
}

resource "artifactory_permission_target" "private" {
name = "devx-generic-thanh-dev"

repo {
  repositories = ["devx-generic-thanh-dev"]

  actions {
    dynamic "groups" {
      for_each = toset([artifactory_group.read_group.name])

      content {
        name        = groups.value
        permissions = ["read"]
      }
    }
  }
}
}

[x] Your version of artifactory: 7.77.11
[x] Your version of terraform: 1.6.6
[x] Your version of terraform provider: 10.7.0

Expected behavior Resources are applied in a single Terraform apply run.

Additional context We run into this issue in our QA and prod environments where we have a 3 node cluster. On dev, this is working without issues.

The issue seemingly started appearing when we added the jfrog/platorm provider to the repo alongside the jfrog/artifactory provider.

TF plan:

Terraform will perform the following actions:

  # module.devx-generic-thanh-dev.artifactory_group.read_group will be created
  + resource "artifactory_group" "read_group" {
      + admin_privileges = false
      + auto_join        = false
      + ***cription      = "Test read group"
      + id               = (known after apply)
      + name             = "artifactory-repo-devx-generic-thanh-dev-read"
      + policy_manager   = false
      + realm            = "ldap"
      + reports_manager  = false
      + watch_manager    = false
    }

  # module.devx-generic-thanh-dev.artifactory_local_generic_repository.generic_repository will be created
  + resource "artifactory_local_generic_repository" "generic_repository" {
      + blacked_out          = false
      + cdn_redirect         = false
      + ***cription          = "Test generic local repository"
      + id                   = (known after apply)
      + inclu***_pattern     = "**/*"
      + key                  = "devx-generic-thanh-dev"
      + package_type         = (known after apply)
      + priority_resolution  = false
      + project_environments = (known after apply)
      + repo_layout_ref      = "simple-default"
      + xray_index           = true
    }

  # module.devx-generic-thanh-dev.artifactory_permission_target.private will be created
  + resource "artifactory_permission_target" "private" {
      + id   = (known after apply)
      + name = "devx-generic-thanh-dev"

      + repo {
          + repositories = [
              + "devx-generic-thanh-dev",
            ]

          + actions {
              + groups {
                  + name        = "artifactory-repo-devx-generic-thanh-dev-read"
                  + permissions = [
                      + "read",
                    ]
                }
            }
        }
    }

Plan: 3 to add, 0 to change, 0 to destroy.

TF apply:

module.devx-generic-thanh-dev.artifactory_group.read_group: Creating...
module.devx-generic-thanh-dev.artifactory_local_generic_repository.generic_repository: Creating...
module.devx-generic-thanh-dev.artifactory_group.read_group: Creation complete after 0s [id=artifactory-repo-devx-generic-thanh-dev-read]
module.devx-generic-thanh-dev.artifactory_permission_target.private: Creating...
module.devx-generic-thanh-dev.artifactory_local_generic_repository.generic_repository: Creation complete after 0s [id=devx-generic-thanh-dev]
╷
│ Warning: Deprecated Resource
│ 
│   with module.devx-generic-thanh-dev.artifactory_permission_target.private,
│   on modules/debug-repo/main.tf line 34, in resource "artifactory_permission_target" "private":
│   34: resource "artifactory_permission_target" "private" {
│ 
│ This resource has been deprecated in favor of "platform_permission"
│ (https://registry.terraform.io/providers/jfrog/platform/latest/docs/resources/permission)
│ resource.
╵
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to
│ module.devx-generic-thanh-dev.artifactory_permission_target.private,
│ provider "provider[\"registry.terraform.io/jfrog/artifactory\"]" produced
│ an unexpected new value: Root object was present, but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

alexhung commented 3 months ago

@tpham-drw Thanks for the reporting. I've added this to our plan to investigate.

alexhung commented 3 months ago

@tpham-drw I noticed something in your HCL. Instead of using a plain text value for the repository (repositories = ["devx-generic-thanh-dev"]), have you tried using a reference?

e.g.

repositories = [artifactory_local_generic_repository. generic_repository.key]

Using the plain text value means Terraform doesn't know artifactory_permission_target is depending on the artifactory_local_generic_repository resource. You can see in the TF output that TF starts creating the repository and before that was completed, it starts on creating the permission target.

The reference value will tell Terraform the dependency and will ensure it won't try to create the permission target until the repository is created first.

As to why this works in dev (single node) vs QA/prod (3 nodes), this is probably because in a single node setup, the instance is (may be?) blocked during repository creating thus results in correct sequencing of resource creation. Whereas in multi-nodes setup, both resource creation API requests can be services simultaneously, thus results in this issue.

At least this is my guess right now.

tpham-drw commented 2 months ago

@alexhung Thanks for the suggestion. I have tested both:

using a reference to the repository and group
adding a lifecycle delay to make sure the repo and group exist before attaching the permission target:

resource "artifactory_local_generic_repository" "generic_repository" {
  key                 = "devx-generic-thanh-dev"
  description         = "Test generic local repository"
  includes_pattern    = "**/*"

  provisioner "local-exec" {
    command = "sleep 5"
  }

  lifecycle {
    ignore_changes = [
      project_key,
    ]
  }
}

resource "artifactory_group" "read_group" {
  name             = "artifactory-repo-devx-generic-thanh-dev-read"
  description      = "Test read group"
  admin_privileges = false
  realm            = "ldap"

  provisioner "local-exec" {
    command = "sleep 5"
  }
}

resource "artifactory_permission_target" "private" {
  name = artifactory_local_generic_repository.generic_repository.key

  repo {
    repositories = [artifactory_local_generic_repository.generic_repository.key]

    actions {
      dynamic "groups" {
        for_each = toset([artifactory_group.read_group.name])

        content {
          name        = groups.value
          permissions = ["read"]
        }
      }
    }
  }
}

This still results in the TF applies failing:


module.devx-generic-thanh-dev3.artifactory_group.read_group: Creating...
module.devx-generic-thanh-dev3.artifactory_local_generic_repository.generic_repository: Creating...
module.devx-generic-thanh-dev3.artifactory_group.read_group: Provisioning with 'local-exec'...
module.devx-generic-thanh-dev3.artifactory_group.read_group (local-exec): Executing: ["/bin/sh" "-c" "sleep 5"]
module.devx-generic-thanh-dev3.artifactory_local_generic_repository.generic_repository: Provisioning with 'local-exec'...
module.devx-generic-thanh-dev3.artifactory_local_generic_repository.generic_repository (local-exec): Executing: ["/bin/sh" "-c" "sleep 5"]
module.devx-generic-thanh-dev3.artifactory_group.read_group: Creation complete after 5s [id=artifactory-repo-devx-generic-thanh-dev3-read]
module.devx-generic-thanh-dev3.artifactory_local_generic_repository.generic_repository: Creation complete after 6s [id=devx-generic-thanh-dev3]
module.devx-generic-thanh-dev3.artifactory_permission_target.private: Creating...
╷
│ Warning: Deprecated Resource
│
│ with module.devx-generic-thanh-dev3.artifactory_permission_target.private,
│ on modules/debug-repo/main.tf line 42, in resource "artifactory_permission_target" "private":
│ 42: resource "artifactory_permission_target" "private" {
│
│ This resource has been deprecated in favor of "platform_permission"
│ ([https://registry.terraform.io/providers/jfrog/platform/latest/docs/resources/permission](https://registry.terraform.io/providers/jfrog/platform/latest/docs/resources/permission))
│ resource.
╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to
│ module.devx-generic-thanh-dev3.artifactory_permission_target.private,
│ provider "provider[\"registry.terraform.io/jfrog/artifactory\"]" produced
│ an unexpected new value: Root object was present, but now absent.
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

I will test if running Terraform against a single node in the cluster will make a difference to the Terraform apply.

alexhung commented 2 months ago

@tpham-drw Thanks for the additional info. There were reports of same error message from other resources but I have not been able to reproduce them. My current suspicion is it has something to do with concurrency but more investigation is required.

tpham-drw commented 1 month ago

Hi @alexhung, I am in the process of testing using a endpoint without load balancing and ran into some issues with LDAP groups needing to be re-created/imported when the provider URL is changed.

Do you know if updating the artifactory_permission_target to platform_permission might help with this issue? It seems unlikely but if it could work then we won't have to put a non-LB workaround.

alexhung commented 1 month ago

@tpham-drw I don't know for sure. I doubt it but it may be worth a try.

jfrog / terraform-provider-artifactory

artifactory_permission_target fails to apply with error: Root object was present, but now absent. #988