Pagination not working?

[Nhoya]

Hi, i was looking at this API while i found that pagination is not really working.. Following the domumentation

to fetch all results for a given query, keep incrementing 'p' until you get a page with an empty results list. Note you can reach further into the results by setting per_page to 100.

but

curl https://searchcode.com/api/codesearch_I/\?q\="soup"\&p\=1000000000000000000000000000000000000000000000000000000000000000000000000000000000000\&per_page\=100

still has results, how is this possible?

just to be clear curl https://searchcode.com/api/codesearch_I/\?q\="soup"\&p\=1000\&per_page\=100 |sha256sum de8ddf66e564a3bf34da710de9fe241fd7faa4e9dc9533fe7cf69a4e68d9a0d8

curl https://searchcode.com/api/codesearch_I/\?q\="soup"\&p\=100\&per_page\=100 |sha256sum de8ddf66e564a3bf34da710de9fe241fd7faa4e9dc9533fe7cf69a4e68d9a0d8 and so on..

P.S. i want to suggest add a last_page to the header (as GitHub does) or something like that, just to make the pagination easier

[boyter]

So it is working as designed, its just poorly described, the documentation is a little out of date. I am in the middle of a large update to this to solve a lot of these issues so it will get better eventually I promise.

Whats happening is that if the page number is greater than 49 (0 based pages) then it is set to 49 that being the last page.

You can use the values page previouspage and nextpage in the JSON response to see this. For the above https://searchcode.com/api/codesearch_I/?q=%22soup%22&p=1000000000000000000000000000000000000000000000000000000000000000000000000000000000000&per_page=100 you will see the values,

"page": 49,
"previouspage": 48,
"nextpage": null,

somewhere in the result. These can be used to determine if there is a next page or not and is a better way to do it.

[Nhoya]

Ok thanks, please consider my suggestion on adding lastpage somewhere in the first answer. This will be useful, for example, to spawn a worker for each page and get the data way faster

[boyter]

To be honest its nice that you don't do that because it saves on compute resources :P

I will add it onto the list of things to add for the next release. You can track it over here https://github.com/boyter/searchcode-server/tree/searchcode with all the code view-able.

Feel free to raise a bug there and ill ensure it makes it into the searchcode move over when it occurs... probably sometime in the months to come.