No support for namespaces (Consul Enterprise)

[8666]

The same policy/token that works for git2consul is not working with gonsul.

I get: TransactionError: {"Results":null,"Errors":[{"OpIndex":0,"What":"Permission denied"}]}

Even a simple single item is not working, DRYRUN produces:

+--+----------------+-------------+------+
|  | OPERATION NAME | CONSUL VERB | PATH |
+--+----------------+-------------+------+
|  | INSERT         | set         | env  |
+--+----------------+-------------+------+

This is the policy

acl = "write"

key_prefix "/" {
  policy = "write"
}   

node_prefix "" {
  policy = "read"
}

session_prefix "" {
  policy = "write"
}

service_prefix "" {
  policy = "write"
  intentions = "write"
}

The exit code is 31

[fabiomsouto]

Hello,

Please run gonsul with --log-level=DEBUG and paste the results here.

Cheers

[8666]

• gonsul --log-level=DEBUG -allow-deletes=true -strategy=DRYRUN -consul-url=https://*.hashicorp.cloud:443 -consul-base-path=data -consul-acl=**** -repo-root=data

[INFO]  [May 14 08:44:37.719] Starting in mode: DRYRUN
[DEBUG] [May 14 08:44:37.719] Starting data retrieve from GIT
[INFO]  [May 14 08:44:37.719] EXPORTER: Skipping Git clone, using local path: data
[DEBUG] [May 14 08:44:37.720] Finished data retrieve from GIT
[DEBUG] [May 14 08:44:37.720] Starting data import to Consul

+--+----------------+-------------+---------------------------------------------------------------------------------------------------------------+
|  | OPERATION NAME | CONSUL VERB |                                                     PATH                                                      |
+--+----------------+-------------+---------------------------------------------------------------------------------------------------------------+
|  | INSERT         | set         | Looks OK, data/.....                                                                                          |
|  | INSERT         | set         | Looks OK                                                                                                      |
|  | INSERT         | set         | Looks OK                                                                                                      |
|  | INSERT         | set         | Looks OK                                                                                                      |
|  | INSERT         | set         | Looks OK                                                                                                      |
|  | INSERT         | set         | Looks OK                                                                                                      |
|  | INSERT         | set         | Looks OK                                                                                                      |
|  | INSERT         | set         | Looks OK                                                                                                      |
|  | INSERT         | set         | Looks OK                                                                                                      |
|  | INSERT         | set         | Looks OK                                                                                                      |
+--+----------------+-------------+---------------------------------------------------------------------------------------------------------------+
[DEBUG] [May 14 08:44:37.775] Finished data import to Consul
[INFO]  [May 14 08:44:37.775] Quitting... bye.

WITH ONCE:

[INFO]  [May 14 08:49:14.144] Starting in mode: ONCE
[DEBUG] [May 14 08:49:14.144] Starting data retrieve from GIT
[INFO]  [May 14 08:49:14.144] EXPORTER: Skipping Git clone, using local path: data
[DEBUG] [May 14 08:49:14.145] Finished data retrieve from GIT
[DEBUG] [May 14 08:49:14.145] Starting data import to Consul
[DEBUG] [May 14 08:49:14.205] CONSUL: creating PUT request
[DEBUG] [May 14 08:49:14.205] CONSUL: calling PUT request
[DEBUG] [May 14 08:49:14.207] CONSUL: reading PUT response
[ERROR] [May 14 08:49:14.207] TransactionError: {"Results":null,"Errors":[{"OpIndex":0,"What":"Permission denied"},{"OpIndex":1,"What":"Permission denied"},{"OpIndex":2,"What":"Permission denied"},{"OpIndex":3,"What":"Permission denied"},{"OpIndex":4,"What":"Permission denied"},{"OpIndex":5,"What":"Permission denied"},{"OpIndex":6,"What":"Permission denied"},{"OpIndex":7,"What":"Permission denied"},{"OpIndex":8,"What":"Permission denied"},{"OpIndex":9,"What":"Permission denied"}]}

I made the acl value in the policy "read" again, should not make any difference. I am using Hashicorp managed consul cluster:

acl = "read"

key_prefix "" {
  policy = "write"
}   

node_prefix "" {
  policy = "read"
}

session_prefix "" {
  policy = "write"
}

service_prefix "" {
  policy = "write"
  intentions = "write"
}

[8666]

@fabiomsouto Any idea?

[fabiomsouto]

This doesn't look like a Gonsul issue, but rather a permissions issue.

What I suggest you to do is to login using the ACL that you've configured for Gonsul, in the Consul dashboard, and try to do the operation manually.

[8666]

I tested it with the same token that Gonsul is using and there are no access issues on the webpage. I can create/modify/delete keys. The token has full permission policy attached to it, but works only for one namespace. (let's say 'development')

Gonsul sees the COnusl values properly, but on write, there are some issues with targeting the namespaces/verification I suppose.

Then I took the master token and Gonsul runs fine with but it works with the default namespace only.

In the managed/enterprise version there are namespaces.. so probably Gonsul is not handling them properly

https://www.consul.io/docs/enterprise/namespaces

Git2consul works just fine but Gonsul has more features.

[8666]

This is the output with the master token that default to 'default' namespace:

[DEBUG] [Jun  1 14:43:56.199] CONSUL: creating PUT request
[DEBUG] [Jun  1 14:43:56.199] CONSUL: calling PUT request
[DEBUG] [Jun  1 14:43:56.204] CONSUL: reading PUT response
[INFO]  [Jun  1 14:43:56.204] Operation: set Path: data/AP-SOUTHEAST-2/Pay/CUST2LIVE/my_event_data.json
[INFO]  [Jun  1 14:43:56.204] Operation: set Path: data/AP-SOUTHEAST-2/Pay/CUSTLIVE/PAYCHANNELCODE.json
[INFO]  [Jun  1 14:43:56.204] Operation: set Path: data/EU-WEST-2/environment-a.json
[INFO]  [Jun  1 14:43:56.204] Finished: 3 Inserts, 0 Updates 0 Deletes
[DEBUG] [Jun  1 14:43:56.204] Finished data import to Consul
[INFO]  [Jun  1 14:43:56.204] Quitting... bye.

and the Hashicorp support can confirm the events on the server side (We do not have direct access to the logs)

With the token for 'development' namespace, there are no log/events on the on the server side, they can not see any errors, like Gonsul was not even trying to do something. However the DRYRUN can find the drift. Probably READ actions are not logged.

@fabiomsouto I am out of ideas.

[eduardev]

Hi @8666

I've had Gonsul working on many production environments before, with and without ACL's and never came across such issue. It seems this is probably related with "ACL Policies Namespace", which looks like an enterprise only feature (never actually had heard of it before), and we can't test against an enterprise version of Consul.

What I would suggest/ask is for you to test your Consul cluster using plain CURL API calls, using your ACL and see how it goes. As soon as you have/find a working CURL with PUT call please reply here with it so we can look into our internal Gonsul calls and try to figure it out from there.

[8666]

There is a 'ns' parameter that is available in the enterprise edition. It works fine if it is omitted or you know the namespace and set it as parameter. Do you force it with value of 'default' ?

These are my results in 'development' namespace with a token that only works in that namespace:

+ curl -s -X PUT https://consul-enterprise-123.hashicorp.cloud/v1/kv/data/test_val -H 'Accept: application/json' -H 'X-Consul-Token: ****' -H 'Content-Type: application/json' --data-raw '{"foo": "zzz"}'
true

+ curl -s -X PUT 'https://consul-enterprise-123.hashicorp.cloud/v1/kv/data/test_val?ns=test' -H 'Accept: application/json' -H 'X-Consul-Token: ****' -H 'Content-Type: application/json' --data-raw '{"foo": "zzz"}'
Permission denied

+ curl -s -X PUT 'https://consul-enterprise-123.hashicorp.cloud/v1/kv/data/test_val?ns=development' -H 'Accept: application/json' -H 'X-Consul-Token: ****' -H 'Content-Type: application/json' --data-raw '{"foo": "zzz"}'
true

+ curl -s -X PUT 'https://consul-enterprise-123.hashicorp.cloud/v1/kv/data/test_val?ns=preprod' -H 'Accept: application/json' -H 'X-Consul-Token: ****' -H 'Content-Type: application/json' --data-raw '{"foo": "zzz"}'
Permission denied

+ curl -s -X PUT 'https://consul-enterprise-123.hashicorp.cloud/v1/kv/data/test_val?ns=production' -H 'Accept: application/json' -H 'X-Consul-Token: ****' -H 'Content-Type: application/json' --data-raw '{"foo": "zzz"}'
Permission denied

+ gonsul --log-level=DEBUG --keep-ext=true --allow-deletes=true -strategy=DRYRUN -consul-url=https://consul-enterprise-123.hashicorp.cloud -consul-base-path=data -consul-acl=**** -repo-root=data
[INFO]  [Jun  2 13:33:11.193] Starting in mode: DRYRUN
[DEBUG] [Jun  2 13:33:11.193] Starting data retrieve from GIT
[INFO]  [Jun  2 13:33:11.193] EXPORTER: Skipping Git clone, using local path: data
[DEBUG] [Jun  2 13:33:11.194] Finished data retrieve from GIT
[DEBUG] [Jun  2 13:33:11.194] Starting data import to Consul

+----+----------------+-------------+-------------------------------------------------------------------------------------------------------------+
|    | OPERATION NAME | CONSUL VERB |                                                    PATH                                                     |
+----+----------------+-------------+-------------------------------------------------------------------------------------------------------------+
...
|    | INSERT         | set         | data/env.txt                                                                                                |
...
| !! | DELETE         | delete      | data/curl                                                                                                   |
| !! | DELETE         | delete      | data/test_val                                                                                               |
+----+----------------+-------------+-------------------------------------------------------------------------------------------------------------+
[DEBUG] [Jun  2 13:33:11.274] Finished data import to Consul
[INFO]  [Jun  2 13:33:11.275] Quitting... bye.

+ gonsul --log-level=DEBUG --keep-ext=true --allow-deletes=true -strategy=ONCE -consul-url=https://consul-enterprise-123.hashicorp.cloud -consul-base-path=data -consul-acl=**** -repo-root=data
[INFO]  [Jun  2 13:33:11.500] Starting in mode: ONCE
[DEBUG] [Jun  2 13:33:11.500] Starting data retrieve from GIT
[INFO]  [Jun  2 13:33:11.500] EXPORTER: Skipping Git clone, using local path: data
[DEBUG] [Jun  2 13:33:11.501] Finished data retrieve from GIT
[DEBUG] [Jun  2 13:33:11.501] Starting data import to Consul
[DEBUG] [Jun  2 13:33:11.609] CONSUL: creating PUT request
[DEBUG] [Jun  2 13:33:11.609] CONSUL: calling PUT request
[DEBUG] [Jun  2 13:33:11.612] CONSUL: reading PUT response
[ERROR] [Jun  2 13:33:11.612] TransactionError: {"Results":null,"Errors":[{"OpIndex":0,"What":"Permission denied"},{"OpIndex":1,"What":"Permission denied"},{"OpIndex":2,"What":"Permission denied"},{"OpIndex":3,"What":"Permission denied"},{"OpIndex":4,"What":"Permission denied"},{"OpIndex":5,"What":"Permission denied"},{"OpIndex":6,"What":"Permission denied"},{"OpIndex":7,"What":"Permission denied"},{"OpIndex":8,"What":"Permission denied"},{"OpIndex":9,"What":"Permission denied"},{"OpIndex":10,"What":"Permission denied"},{"OpIndex":11,"What":"Permission denied"}]}

[8666]

If I use the master token that has full privileges is all namespaces then I can set the same key/value in each namespace with CURL, but I will need to execute the command with different value for parameter 'ns'.

Maybe you can add it as flag to Gonsul?

[eduardev]

@8666 we are not setting that flag anywhere, no. So, a flag like ns added to Gonsul and on each run, if that flag was present, Gonsul would use it on every call. That would work right?

[8666]

@eduardev Looks like that is a working solution, but the command I used with CURL API is a single action..

Looking here: https://www.consul.io/api-docs/txn#namespace if you use transactions it goes as 'Namespace' not 'ns'.

I am available for testing

[eduardev]

@8666 good catch, we always use transactions. Are you able to build Gonsul from source? if you can it would make trial and error easier as we would start a branch with this feature/fix where you could pull-build-test.

[8666]

@eduardev Yes I managed to compile it.

[fabiomsouto]

@8666 thanks for mentioning the namespaces, that explains it.

Currently we don't need namespaces, but we'll keep track of your report as a feature request. If you're feeling up to it, feel free to submit a PR that adds support for namespaces, we'll gladly review and integrate it.

Cheers.

[8666]

I did the implementation, works fine, need to do some more testing and then will create a PR

[8666]

@fabiomsouto The PR is ready, works fine with Enterprise Consul. Please review it, probably nothing else is broken.

[fabiomsouto]

@8666 thanks! We'll review it soon.

[fabiomsouto]

For you reading this, help to test this is appreciated, if you have an enterprise license!

[8666]

For you reading this, help to test this is appreciated, if you have an enterprise license!

I am the same person that submitted the PR code, this is my private account.

[darrenwest-allocate]

I can validate this is working as hoped and that @8866 and @vlatkoboshkoski-allocate are the same person.