Rabbitmq secret engine crashes vault on role read

[mr-miles]

Describe the bug Reading the details of a rabbitmq role from vault appears to crash the process

I am able to add a new role successfully, but reading the details of the role after creation causes strange errors and appears to make the vault container crash and restart.

I found the problem because it was crashing our terraform deployments, however I am able to reproduce it through the http api and the web ui command line.

Comparing the rabbitmq secret backend and the database one, the only difference I can see is e.g.

https://github.com/hashicorp/vault/blob/2acac70160962a61b21f2e1355fa052e464f7f75/builtin/logical/rabbitmq/path_roles.go#L110 - the database plugin constructs a map[string]interface{} rather than structs.New(role).Map(). (this may be my amateur reading of the go code though).

Failing that, my other hunch is that the the interaction between the plugin and the storage is not quite right, so the role creation is storing something that blows up when read back out

Are there any pointers about how the plugins work so I could contribute some useful debugging information?

I can also see that path_roles.go lacks its test file so I will try to poke around there too

To Reproduce Steps to reproduce the behavior:

1. vault secrets enable rabbitmq
2. vault write rabbitmq/config/connection connection_uri=xxx username=xxx password=xxx
3. vault write rabbitmq/roles/newrole vhosts='{"/":{"write": ".", "read": "."}}'
4. vault read rabbitmq/roles/newrole

In the web ui, a redirect is returned causing the command to not return. From the logs it appears the redirect is masking a restart of the container.

Expected behavior

The role details as defined within vault are returned as per the api

Environment: Vault 1.15.1 Backend dynamodb EKS - installed with helm chart 3 node cluster

[hghaf099]

@mr-miles I cannot reproduce the issue. Would you please provide us with a stacktrace of the crash?

[mr-miles]

Sorry for the delay - the investigation was obfuscated by some load balancer setup however I've reproduced it on a one-node cluster.

To reproduce:

• create a rabbtmq backend
• create a role "blah-role"
• $ vault read rabbitmq/creds/blah-role issues credentials without a problem
• $ vault list rabbitmq/roles lists the available roles fine
• $ vault read rabbitmq/roles/blah-role causes the panic

The stacktrace is here:

panic: reflect: reflect.Value.Set using unaddressable value

goroutine 11414 [running]: reflect.flag.mustBeAssignableSlow(0x7bb09a0?) /opt/hostedtoolcache/go/1.21.3/x64/src/reflect/value.go:272 +0x74 reflect.flag.mustBeAssignable(...) /opt/hostedtoolcache/go/1.21.3/x64/src/reflect/value.go:259 reflect.Value.Set({0x7bb0320?, 0xc005a88520?, 0xc005a88290?}, {0x7bb0320?, 0xc003610dc0?, 0xc0060acbd8?}) /opt/hostedtoolcache/go/1.21.3/x64/src/reflect/value.go:2254 +0x65 github.com/hashicorp/vault/audit.(*hashWalker).Primitive(0xc0059d7a40, {0x7bb0320?, 0xc005a88520?, 0x453e6f?}) /home/runner/work/vault/vault/audit/hashstructure.go:384 +0x3d5 github.com/mitchellh/reflectwalk.walkPrimitive(...) /home/runner/go/pkg/mod/ @./reflectwalk.go:270 github.com/mitchellh/reflectwalk.walk({0x7bb0320?, 0xc005a88520?, 0xc0059d7a40?}, {0x96e74c0, 0xc0059d7a40}) /home/runner/go/pkg/mod/ @./reflectwalk.go:197 +0x62f github.com/mitchellh/reflectwalk.walkStruct({0x8c80cc0?, 0xc005a88520?, 0xc0059d7a40?}, {0x96e74c0, 0xc0059d7a40}) /home/runner/go/pkg/mod/ @./reflectwalk.go:404 +0x3c5 github.com/mitchellh/reflectwalk.walk({0x8c80cc0?, 0xc005a88520?, 0xc0059d7a40?}, {0x96e74c0, 0xc0059d7a40}) /home/runner/go/pkg/mod/ @./reflectwalk.go:206 +0x69f github.com/mitchellh/reflectwalk.walkMap({0x8206220?, 0xc005c407e0?, 0xc0059d7a40?}, {0x96e74c0, 0xc0059d7a40}) /home/runner/go/pkg/mod/ @./reflectwalk.go:252 +0x2fa github.com/mitchellh/reflectwalk.walk({0x8206220?, 0xc005c407e0?, 0xc0059d7a40?}, {0x96e74c0, 0xc0059d7a40}) /home/runner/go/pkg/mod/ @./reflectwalk.go:200 +0x58d github.com/mitchellh/reflectwalk.walkMap({0x8206280?, 0xc005c40780?, 0xc0059d7a40?}, {0x96e74c0, 0xc0059d7a40}) /home/runner/go/pkg/mod/ @./reflectwalk.go:252 +0x2fa github.com/mitchellh/reflectwalk.walk({0x8000240?, 0xc003610d10?, 0xc0059d7a40?}, {0x96e74c0, 0xc0059d7a40}) /home/runner/go/pkg/mod/ @./reflectwalk.go:200 +0x58d github.com/mitchellh/reflectwalk.walkMap({0x81ef7a0?, 0xc005c406f0?, 0xc0059d7a40?}, {0x96e74c0, 0xc0059d7a40}) /home/runner/go/pkg/mod/ @./reflectwalk.go:252 +0x2fa github.com/mitchellh/reflectwalk.walk({0x81ef7a0?, 0xc005c406f0?, 0xc0059d7a40?}, {0x96e74c0, 0xc0059d7a40}) /home/runner/go/pkg/mod/ @./reflectwalk.go:200 +0x58d github.com/mitchellh/reflectwalk.Walk({0x81ef7a0?, 0xc005c406f0?}, {0x96e74c0, 0xc0059d7a40}) /home/runner/go/pkg/mod/ @./reflectwalk.go:99 +0x106 github.com/hashicorp/vault/audit.HashStructure(...) /home/runner/work/vault/vault/audit/hashstructure.go:207 github.com/hashicorp/vault/audit.hashMap(0xc0036107a0, 0xc005c406f0?, {0x0, 0x0, 0x0}) /home/runner/work/vault/vault/audit/hashstructure.go:115 +0x205 github.com/hashicorp/vault/audit.HashResponse({0xb8c78d8, 0xc005eb84e0}, {0xb843040?, 0xc003f31340?}, 0xc004825720, 0x0?, {0x0, 0x0, 0x0}, 0x0) /home/runner/work/vault/vault/audit/hashstructure.go:162 +0x366 github.com/hashicorp/vault/audit.(EntryFormatter).FormatResponse(0xc003f2f9f0, {0xb8c78d8, 0xc005eb84e0}, 0xc0060adc10) /home/runner/work/vault/vault/audit/entry_formatter.go:338 +0x2e5 github.com/hashicorp/vault/audit.(EntryFormatter).Process(0xc003f2f9f0, {0xb8c78d8, 0xc005eb84e0}, 0xc005a65740) /home/runner/work/vault/vault/audit/entry_formatter.go:120 +0x405 github.com/hashicorp/eventlogger.(graph).doProcess(0xc0040950e0, {0xb8c78d8, 0xc005eb84e0}, 0xc004071cc0, 0x10?, 0xc003a91da0, 0xc005b38540) **@./graph.go:79 +0xa4 github.com/hashicorp/eventlogger.(graph).process.func1.1({0x412a32?, 0xc0040646f8?}, 0xc003f51968) **@./graph.go:40 +0x88 github.com/hashicorp/eventlogger.(graph).process.func1.(graphMap).Range.func2({0x7bb2660?, 0xc003f7dcc0?}, {0x7859ea0?, 0xc003f51968?}) /home/runner/go/pkg/mod/ @./graphmap.go:28 +0x4d sync.(Map).Range(0xc0040950e0, 0xc0060adf88) /opt/hostedtoolcache/go/1.21.3/x64/src/sync/map.go:476 +0x228 github.com/hashicorp/eventlogger.(graphMap).Range(...) /home/runner/go/pkg/mod/ @./graphmap.go:27 github.com/hashicorp/eventlogger.(graph).process.func1() **@./graph.go:38 +0x9e created by github.com/hashicorp/eventlogger.(graph).process in goroutine 160 **@.***/graph.go:37 +0x156

Vault config is pretty basic:

disable_mlock = true raw_storage_endpoint = true ui = true

storage "dynamodb" { region = "xxx" table = "xxx" ha_enabled = "true" } listener "tcp" { tls_disable = 1 address = "[::]:8200" cluster_address = "[::]:8201" } seal "awskms" { region = "xxx" kms_key_id = "xxx" } service_registration "kubernetes" {}

Not sure where to look next - so any help very much appreciated

Thanks

On Tue, Feb 13, 2024 at 11:58 PM Hamid Ghaf @.***> wrote:

@mr-miles https://github.com/mr-miles I cannot reproduce the issue. Would you please provide us with a stacktrace of the crash?

— Reply to this email directly, view it on GitHub https://github.com/hashicorp/vault/issues/25112#issuecomment-1942873533, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEQD4FP7AUUPGMKVRGL5C3YTP43JAVCNFSM6AAAAABCOOPFM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBSHA3TGNJTGM . You are receiving this because you were mentioned.Message ID: @.***>

[mr-miles]

Also - using /sys/raw I can read a role successfully from the storage. It looks like this:

[mr-miles]

@hghaf099 i added all the details. Mentioning you since it didn’t take the “awaiting response” label away

[divyaac]

This is should have ben fixed by https://github.com/openbao/openbao/issues/97! @mr-miles would you be able to confirm that this fixes the issue?

[mr-miles]

Yes, that openbao patch does fix the problem

On Mon, 19 Aug 2024 at 9:53 PM, divyaac @.***> wrote:

This is should have ben fixed by openbao/openbao#97 https://github.com/openbao/openbao/issues/97! @mr-miles https://github.com/mr-miles would you be able to confirm that this fixes the issue?

— Reply to this email directly, view it on GitHub https://github.com/hashicorp/vault/issues/25112#issuecomment-2297437105, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEQD4DUOGV5VBMOXJO5ICDZSJLM3AVCNFSM6AAAAABCOOPFM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJXGQZTOMJQGU . You are receiving this because you were mentioned.Message ID: @.***>

[divyaac]

Great, thank you @mr-miles! We will close this issue then.

[mr-miles]

@divyaac - which is the vault checkin that fixes this? Might be wrong but AFAICS it is fixed only in openbao but the fix has not been applied to vault so the problem remains.

On Mon, 19 Aug 2024 at 11:35 PM, divyaac @.***> wrote:

Great, thank you @mr-miles https://github.com/mr-miles! We will close this issue then.

— Reply to this email directly, view it on GitHub https://github.com/hashicorp/vault/issues/25112#issuecomment-2297616348, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEQD4BLGPSMPUOVX62Z4WTZSJXLXAVCNFSM6AAAAABCOOPFM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJXGYYTMMZUHA . You are receiving this because you were mentioned.Message ID: @.***>

[divyaac]

Reopening this issue because the openbao fix has not been applied to Vault.