Create a vault /secret/search endpoint - search the list namespace

[TopherGopher]

This endpoint would accept a GET search string and would return a JSON payload of all paths the user is authorized for where 'search string' in path name. There are 2 ways to search:

• Search all secret names/paths for the keyword
• Search the secrets themselves for the keyword For us, we only need the search to really touch the name. There's almost never an instance, for example, where you have a password and you need to search for the secret it belongs to.

We wrote a client-side application that lists all secret directories and secrets, but the amount of recursive requests that you have to do on that end is the bottleneck for our search. Additionally, this loads all authorized secret paths into local active memory, which is...not necessarily ideal...but OK. Moving the work to the server would seem to be the best optimization.

The other feature we could settle on would be a "recursive" flag on the /list endpoint, which would return ALL paths that the user is authorized to see. This would still load the paths all into active memory, but then we would only need to do a single call and could search the entire namespace.

[jktr]

Having deployed vault for consumption by both humans and automation, the lack of search for kv mounts is one of the most immediate pain points users seem to encounter.

As a workaround, we've hacked together a small service that regularly walks an entire kv hierarchy and generates a static html page that lists all paths therein. Each path is a deep link to the respective path in the vault UI, so that users can Ctrl-F on the overview page and proceed directly to vault. While it's not pretty, effectively grants all users the list permission on all paths, and doesn't deal well with churn or a very large number of paths, it does give a semblance of efficient search. User acceptance has been surprisingly high.

[aphorise]

I have a different opinion and do not agree that any search should be provided at all - ever! 🐇 👟 😃

I'm aware of the recursive nature of the issue & scripts required to traverse paths in order to build a complete tree (ie it is possible to hack) - however therein lies several issues.

1. No searching or traversing should ever be facilitated beyond the limited access granted that can be granular and not global and without any facilitated scanning (by design).
2. Absolutely no searching of any secrets - else what's the point of it being a secret if I can search for your IRS number with some wild / known selector?
3. CPU / system resources would be needed to pre-optimise searches trees based on routes that are constant in their change and updating - should Vault do that or focus read write / access / generation of authentication and secrets?
4. Even if searching were presently available in the most rudimentary form how would one limit or deal with recursion (not possible) - especially in cases where there are converged and large listing (millions) or deep nested entries that are longer in their path length than the ending /doc or its secret size.
5. Correlating KVs on policy basis seems like a better alternative to me and therefore limiting any listing that might ever be needed on a per mount basis. Something like Policy -> Path -> Person / Token - and the reset and can be managed for decoupled identities or policies used as reference for deleting all there related mount that are dormant - don't leave secrets around - its not an archive to search!

Now - don't get me wrong you may be able to devise other hacks were you're able to read and store (binary / quick / bubble) search trees based on for example a deciphered storage type (file, etc) that may actually be another access to internal mounts and their logical paths, secrets, etc to search faster / separately offline.

However - IMO Vault should not accommodate for what this issue is requesting or it's intent to get a facilitated search and access to all secrets and every secret including those trees / contents not known to them.

I'd motion to close before the 6th Anniversary of this issue; otherwise I'd bet this will remain open indefinitely for another few years 🤣

[TopherGopher]

@aphorise You have literally dozens of customers which are requesting this feature that disagree with you. All of the points you have brought up have valid technical solutions (again, I refer to LDAP search as a prime example of a technology which has resolved a majority of these issues). Technical issues are resolvable. We could cache all paths and do a text-based search across them for example. We could provide parameters to filter the namespace of the search. That all being said, I too think this ticket should be closed as it's being obtusely ignored by those which could make a difference. Most customers want it. The engineers that work on it don't want it and won't listen to the customers. I suggest all on this thread find other solutions for storage should they want to search them and stop paying Hashicorp when they refuse to even consider the possibility of a limited search endpoint. We went to using AWS secrets manager instead and totally ditched vault. AWS may use vault under the covers, not sure, but on top, they provide a basic search of all the paths. and that's all we needed. So - if the technology won't adapt, then users will find other solutions, and things like this will happen to the revenue.

And yes, I realize I'm a little salty, but I think after 5 years of trying to get an issue worked and getting stonewalled deserves some salt. Closing this issue.

[FCTN-RRaitz]

As @TopherGopher pointed out, my organization has moved on and found other solutions and placed Vault on the list of "Does not meet requirements for consideration"

[heatherezell]

I am going to lock comments on this issue due to comments becoming heated.