This PR implements "Approach 2: Next method" as mentioned in the above issue. Main reasons are that it supports callbacks and it can be used to create an approach similar to Approach 3.
Note that this PR doesn't cover pagination support for getJsonBySearchId.
Tests
I've added tests for 6 of the 7 types of pagination approaches:
Offset only: google_maps
Page only: apple_reviews
Offset + size: baidu
Page + size: ebay
Offset + page + size: home_depot
Token only: google_scholar_profiles
Offset + page: The only engine that uses this approach, google_product, relies on an approach that is similar to the token only approach.
Additional notes
Pagination using the .next() method is currently only supported by engines that respond with a serpapi_pagination.next property.
E.g. google_jobs allows for pagination via the start offset param, but currently does not return a serpapi_pagination.next property. For these cases, you need to send in the start offset param manually.
I couldn't replicate it for Ebay, but nonetheless, will keep it in as a safeguard.
I found that duckduckgo has a similar behaviour and have added tests for it.
For duckduckgo, there's a quirk as it has a default parameter. This causes one extra call to be made. Notice the initial params are { q: "coffee", start: 30 } but the next params extracted from page1 are { kl: "us-en", q: "coffee", start: 30}. This difference means that page1.next is returned. It's only after obtaining page2 that the current params (which includes kl) and next params match, implying there's no next page.
Sometimes duckduckgo oscillates between different next parameters:
Handles #2
This PR implements "Approach 2: Next method" as mentioned in the above issue. Main reasons are that it supports callbacks and it can be used to create an approach similar to Approach 3.
Note that this PR doesn't cover pagination support for
getJsonBySearchId
.Tests
I've added tests for 6 of the 7 types of pagination approaches:
google_maps
apple_reviews
baidu
ebay
home_depot
google_scholar_profiles
Offset + page: The only engine that uses this approach,google_product
, relies on an approach that is similar to thetoken
only approach.Additional notes
Pagination using the
.next()
method is currently only supported by engines that respond with aserpapi_pagination.next
property.google_jobs
allows for pagination via thestart
offset param, but currently does not return aserpapi_pagination.next
property. For these cases, you need to send in thestart
offset param manually.pagination.next
is used instead. This is for thegoogle_scholar_profiles
engine.Although
yahoo_shopping
returns aserpapi_pagination.next
property, I've excluded it as there are currently some issues with itThere is a check if the next parameters are equal to the current parameters. This check is based on this issue: https://github.com/serpapi/public-roadmap/issues/144.
I couldn't replicate it for Ebay, but nonetheless, will keep it in as a safeguard.{ q: "coffee", start: 30 }
but the next params extracted frompage1
are{ kl: "us-en", q: "coffee", start: 30}
. This difference means thatpage1.next
is returned. It's only after obtainingpage2
that the current params (which includeskl
) and next params match, implying there's no next page.