Consul configs on secondary dc for federated datacenters - 1.9.3

[pvyaka01]

Overview of the Issue

"consul config read" in secondary dc is not listing the config entries although we can write the configurations

Am I doing something wrong?

Reproduction Steps

1. Federate clusters with ACL replication enabled (https://learn.hashicorp.com/tutorials/consul/access-control-replication-multiple-datacenters#create-the-replication-token-for-acl-management)

2. Connect services across DC (https://learn.hashicorp.com/tutorials/consul/service-mesh-gateways)

3. Create hcl to write service-defaults on secondary dc Sample: webk8s.hcl "Kind"= "service-defaults" "Name"= "webk8s" "Protocol"= "http"

4. Apply the configuration on secondary dc consul config write webk8s.hcl Config entry written: service-defaults/webk8s

5. Try reading the configuration on secondary dc consul config read -kind service-defaults -name webk8s Error reading config entry service-defaults/webk8s: Unexpected response code: 404 (Config entry not found for "service-defaults" / "webk8s")

6. Try reading the configuration on primary dc consul config read -kind service-defaults -name webk8s Response: { "Kind": "service-defaults", "Name": "webk8s", "Protocol": "http", "MeshGateway": {}, "Expose": {}, "CreateIndex": 138153833, "ModifyIndex": 138476514 }

7. Try to list all service-defaults configuration entries on secondary dc consul config list -kind service-defaults Note that it doesn't list any

8. Try to list all service-defaults configuration entries on primary dc consul config list -kind service-defaults webk8s

Consul info for both Client and Server

Server info -

``` agent: check_monitors = 7 check_ttls = 0 checks = 16 services = 16 build: prerelease = revision = f55da930 version = 1.9.3 consul: acl = enabled bootstrap = false known_datacenters = 2 leader = false leader_addr = x.x.x.x:8300 server = true raft: applied_index = 138477672 commit_index = 138477672 fsm_pending = 0 last_contact = 46.524077ms last_log_index = 138477672 last_log_term = 15292 last_snapshot_index = 138467023 last_snapshot_term = 14653 latest_configuration = [{Suffrage:Voter ID:c3fd1fd0-84cd-845f-bd53-55abb2625bd9 Address:x.x.x.x:8300} {Suffrage:Voter ID:cbe153a0-94ae-37f6-b187-e0c4c2c7ae6d Address:x.x.x.x:8300} {Suffrage:Voter ID:891a90da-b9e1-e539-fe14-bf7074820914 Address:x.x.x.x:8300}] latest_configuration_index = 0 num_peers = 2 protocol_version = 3 protocol_version_max = 3 protocol_version_min = 0 snapshot_version_max = 1 snapshot_version_min = 0 state = Follower term = 15292 runtime: arch = amd64 cpu_count = 4 goroutines = 2352 max_procs = 4 os = linux version = go1.15.6 serf_lan: coordinate_resets = 0 encrypted = true event_queue = 0 event_time = 422 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 116342 members = 361 query_queue = 0 query_time = 9 serf_wan: coordinate_resets = 0 encrypted = true event_queue = 0 event_time = 1 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 9177 members = 6 query_queue = 0 query_time = 9 ```

Operating system and Environment details

Red Hat Enterprise Linux Server release 7.9 (Maipo)

Log Fragments

No errors in logs related to configs

[pvyaka01]

Any update about this or am I doing something wrong with the setup?

[aoskotsky-amplify]

I am also seeing this issue. Are there any solutions for this?

[spuder]

We also ran into this when using terraform + consul.

Update: Workaround isn't working

Workaround

Revert all consul_config_entry resources to consul_intention with terraform

resource "consul_config_entry" "foo" {
  name = "foo"
  kind = "service-intentions"
  config_json = jsonencode({
    Sources = [
      {
        Name       = "bar"
        Action     = "allow"
        Type       = "consul"
        Precedence = 9

      }
    ]
  })
}

resource "consul_intention" "foo" {
  source_name      = "bar"
  destination_name = "foo"
  action           = "allow"
}

[Amier3]

Hey @spuder

We think this issue is similar to #9319 which was fixed in PR #12307 . Could you confirm if you're still experiencing this? 👍