search

Cgboal / SonarSearch

A rapid API for the Project Sonar dataset
MIT License
641 stars 96 forks source link

Configurable limit for HTTP APIs and pagination indicator #21

Open nirvana-msu opened 3 years ago

nirvana-msu commented 3 years ago

Thank you for the valuable tool!

A few suggestions on improvements for HTTP API:

1) Allow user to specify the limit (when we don't need the whole 10k - currently hardcoded limit). It could still be capped at 10k to prevent abuse. 2) Provide indication in the response whether "next page" exists

Both of these would allow to query the service in a more considerate manner, which should also help reduce the load.

If possible, it would also be helpful if total count could be included in any paginated response.

Cgboal commented 3 years ago

Doing a count to see how many rows there are in total can take quite a while on some queries which is why it wasn't included in the response. Doing dynamic limits should be doable though.

nirvana-msu commented 3 years ago

Doing a count to see how many rows there are in total can take quite

Yeah, that's why I added a suggestion to add an indicator of whether "next page" exists - that would at least tell users if they should make another query if they need all results.

Thanks!

Cgboal commented 3 years ago

Try the gRPC API instead, you don't need to bother with pagination there as it streams the results.