search

hashicorp / consul

Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.
https://www.consul.io
Other
28.24k stars 4.41k forks source link

Implement general-purpose pagination functionality in KV API #2168

Open josiahhaswell opened 8 years ago

josiahhaswell commented 8 years ago

Key hierarchies can grow to be quite large,and this introduces unnecessary and undesirable latencies during retrieval. It would be very nice if the KV API supported pagination. This could be done in a backwards-compatible fashion via query params (possibly something along the lines of)

/v1/kv/sample-key?start=0&end=10

Ordering would also generally be useful; adding 

/v1/kv/sample-key?start=0&end=10&order=desc|asc

would sort the results by the lexicographical ordering of the keys under sample-key.

This could also be integrated consistently with recurse by performing a depth-first traversal of the keys in range [start, end] under the provided sort criterion and returning that set.

Thanks! Josiah

MatthiasGrandl commented 3 years ago

Any news on this? It seems incredible that such a feature does not exist while filtering does.

vnktram commented 2 years ago

Can I pick this one?

jkirschner-hashicorp commented 2 years ago

Hi @vnktram,

Thanks for volunteering to contribute to Consul! Give me a bit of time to seek the Consul team's perspective on this first - I'll update you next week.

dnephin commented 2 years ago

Thank you for your interest in implementing this feature! I think before we review any implementation it would be good to outline how the problem will be solved. Implementing pagination in Consul is more difficult than it might be in other systems.

One of the challenges is that Consul does not use sequential IDs. The KV entries are indexed by the key name. The keys could change between requests, which could lead to duplicate or missing keys in the response. One option for solving this problem might be to use the CreateIndex as the ID when querying with pagination. That should prevent updates to existing keys from changing the results.

Another challenge is how to query the state store. We currently do not index by CreateIndex, and we do not use range queries (as far as I know). So we would need to add an index to the state store. We would also need to use https://pkg.go.dev/github.com/hashicorp/go-memdb#Txn.LowerBound to query the appropriate range of values (instead of Txn.Get).

If you are interested in picking up this issue, we'd encourage you to first write up a design for the solution (either as a comment in this issue, or as a new issue). The design should cover how the above two challenges will be addressed, and also detail how the new HTTP API query parameters will work, and if there is any interaction with existing API query parameters. That write up will make it much easier for us review any pull requests that implement this feature.

vnktram commented 2 years ago

@dnephin Ack. Let me document the proposed design with your suggestions in mind and share it shortly.

mstrYoda commented 4 weeks ago

Hi all, what is the current status of this issue?

@vnktram @dnephin ?