[TT-13227] Add support for subdirectories when using Vault data store

[vanny96]

Describe the problem As described here, the vault data storage seems to have an inconsistent behaviour depending on where it is used.

In particular the behaviour for "any string field" seems to be quite limited, as it enforces all the key/value pairs to be set in the /secret/tyk-apis Vault secret (btw, the documentation doesn't mention the requirement for the Secret Engine to be named "secret", without looking at the source code there would be no way of knowing.)

Describe the solution you'd like Ideally, the behaviour for "any string field" would be exactly the same as for the other fields (listenPath, upstream, configs and so on). If that's not possible due to back compatibility issues, support for subdirectories should be added so that:

• vault://KEY still resolves to /secrets/data/tyk-apis.KEY
• vault://path.KEY resolves to /secrets/data/tyk-apis/<path>.KEY

Describe alternatives you've considered I think on the short term the subdirectory approach is the easiest to introduce as it won't break back-compatibility, but a more robust implementation that behaves as the configuration files would be heavily preferred.

For the back-compat approach, I created a draft PR with a rough idea of how it could be implemented. You can see more here: https://github.com/TykTechnologies/tyk/pull/6588

For the preferred approach, we could have a replaceVaultSecretsV2 function which uses a better resolution method and could be opt-in to begin with, and eventually become the default approach.

[andyo-tyk]

Hi @vanny96,

Thank you for this suggestion. It's true that the initial implementation of "any string field" has known limitations and was released as an MVP. Your feedback helps to confirm that our plans to extend the functionality further are required.

Your PR is greatly appreciated and we will consider this as part of our work to enhance the Vault key-value storage.

Thanks again for your contribution.