search

hashicorp / consul

Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.
https://www.consul.io
Other
28.37k stars 4.42k forks source link

API endpoint does not parse correctly complex JSON rules #4503

Closed aparedero closed 6 years ago

aparedero commented 6 years ago

Overview of the Issue

API endpoint does not parse correctly complex JSON rules. I'm trying to create a new ACL via curl with management token sending payload in json format. Creating the token trough the UI interface loads JSON syntax successfully and works (eg: Using mgmt token switching key policy from write to deny allows and disallows creating new K/V registers

Using the sample-payload works

$ cat sample-payload.json
{
  "Name": "my-app-token",
  "Type": "client",
  "Rules": ""
}

$ curl --header "X-Consul-Token: 1234ad12-b665-5c40-ae10-7f17d4d68471" --request PUT     --data @sample-payload.json http://consul.mycompany.internal/v1/acl/create
{"ID":"169f9d72-f41f-0f6c-7f0b-3c3cf1632959"}
 $

Reproduction Steps

Steps to reproduce this issue, eg:

  1. Create a cluster with 5 server nodes

  2. Create a JSON file called payload an copy the following contents:

    "ID": "12341234-1234-1234-1234-1234bf00c66e",
"Name": "aaa_server_no_management",
"Type": "client",
"Rules": [
          {
            "agent": [
                {
                    "": [
                        {
                            "policy": "write"
                        }
                    ]
                }
            ],
            "event": [
                {
                    "": [
                        {
                            "policy": "read"
                        }
                    ]
                }
            ],
            "key": [
                {
                    "": [
                        {
                            "policy": "write"
                        }
                    ]
                },
                {
                    "vault": [
                        {
                            "policy": "deny"
                        }
                    ]
                }
            ],
            "keyring": "deny",
            "node": [
                {
                    "": [
                        {
                            "policy": "read"
                        }
                    ]
                }
            ],
            "operator": "read",
            "query": [
                {
                    "": [
                        {
                            "policy": "read"
                        }
                    ]
                }
            ],
            "service": [
                {
                    "": [
                        {
                            "policy": "write"
                        }
                    ]
                }
            ]
        }
      ]
}

  3. Execute curl command

    curl \
--request PUT \
--data @payload.json \
http://consul.mycompany.internal/v1/acl/create

  4. View error

    Request decode failed: 1 error(s) decoding:\n\n* 'Rules' expected type 'string', got unconvertible type '[]interface {}

Consul info for both Client and Server

Server info ``` / # consul info agent: check_monitors = 0 check_ttls = 0 checks = 0 services = 0 build: prerelease = revision = e716d1b5 version = 1.2.2 consul: bootstrap = false known_datacenters = 1 leader = true leader_addr = 10.1.2.177:8300 server = true raft: applied_index = 209 commit_index = 209 fsm_pending = 0 last_contact = 0 last_log_index = 209 last_log_term = 2 last_snapshot_index = 0 last_snapshot_term = 0 latest_configuration = [{Suffrage:Voter ID:4a1ca5df-5941-5273-ee54-669c3ffbcf91 Address:10.1.2.70:8300} {Suffrage:Voter ID:d6b10a67-1c9f-e559-d717-cd77f6e8ca7f Address:10.1.1.191:8300} {Suffrage:Voter ID:06a18810-07e2-7212-595c-a97d0af11c9e Address:10.1.1.160:8300} {Suffrage:Voter ID:443f6232-bd1a-3b23-00f6-768b3b42dfdd Address:10.1.2.231:8300} {Suffrage:Voter ID:b6666b3e-779a-251c-a0d8-3613d8c11bef Address:10.1.2.177:8300}] latest_configuration_index = 1 num_peers = 4 protocol_version = 3 protocol_version_max = 3 protocol_version_min = 0 snapshot_version_max = 1 snapshot_version_min = 0 state = Leader term = 2 runtime: arch = amd64 cpu_count = 2 goroutines = 125 max_procs = 2 os = linux version = go1.10.1 serf_lan: coordinate_resets = 0 encrypted = true event_queue = 0 event_time = 2 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 5 members = 5 query_queue = 0 query_time = 1 serf_wan: coordinate_resets = 0 encrypted = true event_queue = 0 event_time = 1 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 8 members = 5 query_queue = 0 query_time = 1 ```

Operating system and Environment details

Consul 1.2.2 in Docker using Amazon EKS platform. Using traefik as L7 balancer.

Log Fragments

There is no information relevant about this. It appears a normal startup

Log ``` 2018/08/08 11:44:10 [INFO] raft: Initial configuration (index=0): [] 2018/08/08 11:44:10 [INFO] raft: Node at 10.1.2.177:8300 [Follower] entering Follower state (Leader: "") 2018/08/08 11:44:10 [INFO] serf: EventMemberJoin: test-consul-0.test-consul 10.1.2.177 2018/08/08 11:44:10 [INFO] serf: EventMemberJoin: test-consul-0 10.1.2.177 2018/08/08 11:44:10 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp) 2018/08/08 11:44:10 [INFO] consul: Adding LAN server test-consul-0 (Addr: tcp/10.1.2.177:8300) (DC: test-consul) 2018/08/08 11:44:10 [INFO] consul: Handled member-join event for server "test-consul-0.test-consul" in area "wan" 2018/08/08 11:44:10 [WARN] agent/proxy: running as root, will not start managed proxies 2018/08/08 11:44:10 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp) 2018/08/08 11:44:10 [INFO] agent: Started HTTP server on [::]:8500 (tcp) 2018/08/08 11:44:10 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce os packet scaleway softlayer triton vsphere 2018/08/08 11:44:10 [INFO] agent: Joining LAN cluster... 2018/08/08 11:44:10 [INFO] agent: (LAN) joining: [test-consul-1.test-consul.default.svc test-consul-2.test-consul.default.svc test-consul-3.test-consul.default.svc test-consul-4.test-consul.default.svc] 2018/08/08 11:44:10 [INFO] serf: EventMemberJoin: test-consul-2 10.1.2.70 2018/08/08 11:44:10 [INFO] consul: Adding LAN server test-consul-2 (Addr: tcp/10.1.2.70:8300) (DC: test-consul) 2018/08/08 11:44:10 [INFO] agent: started state syncer 2018/08/08 11:44:10 [INFO] serf: EventMemberJoin: test-consul-2.test-consul 10.1.2.70 2018/08/08 11:44:10 [INFO] consul: Handled member-join event for server "test-consul-2.test-consul" in area "wan" 2018/08/08 11:44:10 [INFO] serf: EventMemberJoin: test-consul-3 10.1.1.191 2018/08/08 11:44:10 [INFO] serf: EventMemberJoin: test-consul-1 10.1.1.160 2018/08/08 11:44:10 [INFO] serf: EventMemberJoin: test-consul-4 10.1.2.231 2018/08/08 11:44:10 [INFO] consul: Adding LAN server test-consul-3 (Addr: tcp/10.1.1.191:8300) (DC: test-consul) 2018/08/08 11:44:10 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.1.2.70:8300,10.1.1.191:8300,10.1.1.160:8300,10.1.2.231:8300,10.1.2.177:8300 2018/08/08 11:44:10 [INFO] consul: Adding LAN server test-consul-1 (Addr: tcp/10.1.1.160:8300) (DC: test-consul) 2018/08/08 11:44:10 [INFO] consul: Adding LAN server test-consul-4 (Addr: tcp/10.1.2.231:8300) (DC: test-consul) 2018/08/08 11:44:10 [INFO] serf: EventMemberJoin: test-consul-1.test-consul 10.1.1.160 2018/08/08 11:44:10 [INFO] serf: EventMemberJoin: test-consul-3.test-consul 10.1.1.191 2018/08/08 11:44:10 [INFO] serf: EventMemberJoin: test-consul-4.test-consul 10.1.2.231 2018/08/08 11:44:10 [INFO] consul: Handled member-join event for server "test-consul-1.test-consul" in area "wan" 2018/08/08 11:44:10 [INFO] consul: Handled member-join event for server "test-consul-3.test-consul" in area "wan" 2018/08/08 11:44:10 [INFO] consul: Handled member-join event for server "test-consul-4.test-consul" in area "wan" 2018/08/08 11:44:10 [INFO] agent: (LAN) joined: 4 Err: 2018/08/08 11:44:10 [INFO] agent: Join LAN completed. Synced with 4 initial agents 2018/08/08 11:44:17 [WARN] raft: Heartbeat timeout from "" reached, starting election 2018/08/08 11:44:17 [INFO] raft: Node at 10.1.2.177:8300 [Candidate] entering Candidate state in term 2 2018/08/08 11:44:17 [INFO] raft: Election won. Tally: 3 2018/08/08 11:44:17 [INFO] raft: Node at 10.1.2.177:8300 [Leader] entering Leader state 2018/08/08 11:44:17 [INFO] raft: Added peer 4a1ca5df-5941-5273-ee54-669c3ffbcf91, starting replication 2018/08/08 11:44:17 [INFO] raft: Added peer d6b10a67-1c9f-e559-d717-cd77f6e8ca7f, starting replication 2018/08/08 11:44:17 [INFO] raft: Added peer 06a18810-07e2-7212-595c-a97d0af11c9e, starting replication 2018/08/08 11:44:17 [INFO] raft: Added peer 443f6232-bd1a-3b23-00f6-768b3b42dfdd, starting replication 2018/08/08 11:44:17 [INFO] consul: cluster leadership acquired 2018/08/08 11:44:17 [INFO] consul: New leader elected: test-consul-0 2018/08/08 11:44:17 [WARN] raft: AppendEntries to {Voter 4a1ca5df-5941-5273-ee54-669c3ffbcf91 10.1.2.70:8300} rejected, sending older logs (next: 1) 2018/08/08 11:44:17 [WARN] raft: AppendEntries to {Voter 443f6232-bd1a-3b23-00f6-768b3b42dfdd 10.1.2.231:8300} rejected, sending older logs (next: 1) 2018/08/08 11:44:17 [INFO] raft: pipelining replication to peer {Voter 06a18810-07e2-7212-595c-a97d0af11c9e 10.1.1.160:8300} 2018/08/08 11:44:17 [INFO] raft: pipelining replication to peer {Voter 4a1ca5df-5941-5273-ee54-669c3ffbcf91 10.1.2.70:8300} 2018/08/08 11:44:17 [INFO] raft: pipelining replication to peer {Voter 443f6232-bd1a-3b23-00f6-768b3b42dfdd 10.1.2.231:8300} 2018/08/08 11:44:17 [INFO] consul: Created ACL master token from configuration 2018/08/08 11:44:17 [INFO] consul: ACL bootstrap disabled, existing management tokens found 2018/08/08 11:44:17 [INFO] consul: member 'test-consul-4' joined, marking health alive 2018/08/08 11:44:17 [INFO] consul: member 'test-consul-0' joined, marking health alive 2018/08/08 11:44:17 [INFO] consul: member 'test-consul-2' joined, marking health alive 2018/08/08 11:44:17 [WARN] raft: AppendEntries to {Voter d6b10a67-1c9f-e559-d717-cd77f6e8ca7f 10.1.1.191:8300} rejected, sending older logs (next: 1) 2018/08/08 11:44:17 [INFO] consul: member 'test-consul-3' joined, marking health alive 2018/08/08 11:44:17 [INFO] raft: pipelining replication to peer {Voter d6b10a67-1c9f-e559-d717-cd77f6e8ca7f 10.1.1.191:8300} 2018/08/08 11:44:17 [INFO] consul: member 'test-consul-1' joined, marking health alive 2018/08/08 11:44:17 [INFO] agent: Synced node info illegal ``` Use `-log-level=TRACE` on the client and server to capture the maximum log detail.
aparedero commented 6 years ago

Update: Writing payload in HCL format scaping quotes and new lines it seems to work.

{
    "ID": "12341234-1234-1234-1234-1234bf00c66e",
    "Name": "aaa_server_no_management",
    "Type": "client",
    "Rules": "agent \"\" {\n   policy = \"write\"\n }\n key \"\" {\n   policy = \"write\"\n }\n key \"vault\" {\n   policy = \"deny\"\n }\n event \"\" {\n   policy = \"read\"\n }\n keyring = \"deny\"\n node \"\" {\n   policy = \"read\"\n }\n operator = \"read\"\n query \"\" {\n   policy = \"read\"\n }\n service \"\" {\n   policy = \"write\"\n }"
}
pearkes commented 6 years ago

I think your JSON format is not to the specification. The error message there is saying that it can't convert an array, but expected a string, and this matches what we recommend in the rule specification. The HCL specification you posted looks correct. I would revisit the ACL guide and modify the format of the JSON document and it should work as expected!