Better support for pagination

[sebastianquek]

Goals

This module should make handling pagination simpler across all engines.

Context

Currently, pagination needs to be handled manually. One approach is the following:

import { config, getJson } from "serpapi";

config.api_key = process.env.API_KEY;

const num = 10; // Number of results per page
let start = 0; // Results offset

const links = [];

while (start < 50) { // Get up to 50 results
  const json = await getJson("google", {
    q: "coffee",
    location: "Austin, Texas",
    start,
    num,
  });
  const pageLinks = json["organic_results"].map((r) => r.link);
  links.push(...pageLinks);
  start += num;
}

console.log(links);

This works for engines that support the fetching of results by an offset + size. For example,

• Google - start + num
• Bing - first + count
• Baidu - pn + rn

However, not all engines rely on this offset + size concept. For example,

• Ebay - _pgn refers to the page number and not the results offset.
• Google Play Store - next_page_token is used as a token for the next page.
• Some others have been mentioned here: https://github.com/serpapi/google-search-results-python/issues/22

For these less common approaches, users will need to be aware of it and update their code accordingly.

Full list of engines that support pagination

There are 7 types:

Offset only, e.g. google_jobs, yahoo

Engine | Param | Type | Description -- | -- | -- | -- google_jobs | start | number | Parameter defines the result offset. It skips the given number of results. It’s used for pagination. (e.g., 0 (default) is the first page of results, 10 is the 2nd page of results, 20 is the 3rd page of results, etc.). google_reverse_image | start | number | Parameter defines the result offset. It skips the given number of results. It’s used for pagination. (e.g., 0 (default) is the first page of results, 10 is the 2nd page of results, 20 is the 3rd page of results, etc.). google_maps | start | number | Parameter defines the result offset. It skips the given number of results. It’s used for pagination. (e.g., 0 (default) is the first page of results, 20 is the 2nd page of results, 40 is the 3rd page of results, etc.). google_events | start | number | Parameter defines the result offset. It skips the given number of results. It’s used for pagination. (e.g., 0 (default) is the first page of results, 10 is the 2nd page of results, 20 is the 3rd page of results, etc.). yahoo | b | string | Parameter defines the result offset. It skips the given number of results. It’s used for pagination. (e.g., 1 (default) is the first page of results, 11 is the 2nd page of results, 21 is the 3rd page of results, etc.). yahoo_images | b | string | Parameter defines the result offset. It skips the given number of results. It’s used for pagination. (e.g., 1 (default) starts from the first result, 61 starts from the 61st result, 121 starts from the 121st result, etc.). yahoo_videos | b | string | Parameter defines the result offset. It skips the given number of results. It’s used for pagination. (e.g., 1 (default) starts from the first result, 61 starts from the 61st result, 121 starts from the 121st result, etc.). duckduckgo | start | number | Parameter defines the result offset. It skips the given number of results. It’s used for pagination. When pagination is not being used (initial search request), number of organic_results can vary between 26 and 30. When pagination is being used (value of start parameter is bigger then 0), organic_results return 50 results. yelp | start | number | Parameter defines the result offset. It skips the given number of results. It’s used for pagination. (e.g., 0 (default) is the first page of results, 10 is the 2nd page of results, 20 is the 3rd page of results, etc.). yelp_reviews | start | number | Parameter defines the result offset. It skips the given number of results. It’s used for pagination. (e.g., 0 (default) is the first page of results, 10 is the 2nd page of results, 20 is the 3rd page of results, etc.).

Page only, e.g. yandex, apple_reviews

Engine | Param | Type | Description -- | -- | -- | -- google (google images only) | ijn | string | Parameter defines the page number for Google Images. There are 100 images per page. This parameter is equivalent to start (offset) = ijn * 100. This parameter works only for Google Images (set tbm to isch). yandex | p | string | Parameter defines page number. Pagination starts from 0. yandex_images | p | string | Parameter defines the page number. Pagination starts from 0, and it can return up to 30 results. yandex_videos | p | string | Parameter defines the page number. Pagination starts from 0, and it can return up to 30 results. walmart_product_reviews | page | string | Value is used to get the reviews on a specific page. (e.g., 1 (default) is the first page of results, 2 is the 2nd page of results, 3 is the 3rd page of results, etc.). apple_reviews | page | string | Parameter is used to get the items on a specific page. (e.g., 1 (default) is the first page of results, 2 is the 2nd page of results, 3 is the 3rd page of results, etc.).

Offset + size, e.g. google, bing, baidu

Engine | Param | Type | Description -- | -- | -- | -- google | start | number | Parameter defines the result offset. It skips the given number of results. It’s used for pagination. (e.g., 0 (default) is the first page of results, 10 is the 2nd page of results, 20 is the 3rd page of results, etc.). google | num | string | Parameter defines the maximum number of results to return. (e.g., 10 (default) returns 10 results, 40 returns 40 results, and 100 returns 100 results). google_scholar | start | number | Parameter defines the result offset. It skips the given number of results. It’s used for pagination. (e.g., 0 (default) is the first page of results, 10 is the 2nd page of results, 20 is the 3rd page of results, etc.). google_scholar | num | string | Parameter defines the maximum number of results to return, limited to 20. (e.g., 10 (default) returns 10 results, 20 returns 20 results). google_scholar_author | start | number | Parameter defines the result offset. It skips the given number of results. It’s used for pagination. (e.g., 0 (default) is the first page of results, 20 is the 2nd page of results, 40 is the 3rd page of results, etc.). google_scholar_author | num | string | Parameter defines the number of results to return. (e.g., 20 (default) returns 20 results, 40 returns 40 results, etc.). Maximum number of results to return is 100. bing | first | string | Parameter controls the offset of the organic results. This parameter defaults to 1. (e.g., first=10 will move the 10th organic result to the first position). bing | count | string | Parameter controls the number of results per page. Minimum: 1, Maximum: 50. This parameter is only a suggestion and might not reflect actual results returned. bing_news | first | string | Parameter controls the offset of the organic results. This parameter defaults to 1. (e.g., first=10 will move the 10th organic result to the first position). bing_news | count | string | Parameter controls the number of results per page. This parameter is only a suggestion and might not reflect actual results returned. bing_images | first | string | Parameter controls the offset of the organic results. This parameter defaults to 1. (e.g., first=10 will move the 10th organic result to the first position). bing_images | count | string | Parameter controls the number of results per page. This parameter is only a suggestion and might not reflect the returned results. baidu | pn | string | Parameter defines the result offset. It skips the given number of results. It’s used for pagination. (e.g., 0 (default) is the first page of results, 10 is the 2nd page of results, 20 is the 3rd page of results, etc.). baidu | rn | string | Parameter defines the maximum number of results to return, limited to 50. (e.g., 10 (default) returns 10 results, 30 returns 30 results, and 50 returns 50 results). This parameter is only available for desktop and tablet searches. baidu_news | pn | string | Parameter defines the result offset. It skips the given number of results. It’s used for pagination. (e.g., 0 (default) is the first page of results, 10 is the 2nd page of results, 20 is the 3rd page of results, etc.). baidu_news | rn | string | Parameter defines the maximum number of results to return, limited to 50. (e.g., 10 (default) returns 10 results, 30 returns 30 results, and 50 returns 50 results).

Offset + page, e.g. google_product

Engine | Param | Type | Description -- | -- | -- | -- google_product | start | number | Parameter defines the result offset. It skips the given number of results. It’s used for pagination. (e.g., 0 (default) is the first page of results, 10 is the 2nd page of results, 20 is the 3rd page of results, etc.) This parameter works only for Google Online Sellers and Reviews. google_product | page | string | Parameter defines the page number for Google Online Sellers and Reviews. There are 10 results per page. This parameter is equivalent to start (offset) = page * 10. This parameter works only for Google Online Sellers and Reviews.

Page + size, e.g. ebay, walmart

Engine | Param | Type | Description -- | -- | -- | -- ebay | _pgn | string | Parameter defines the page number. It’s used for pagination. (e.g., 1 (default) is the first page of results, 2 is the 2nd page of results, 3 is the 3rd page of results, etc.). ebay | _ipg | string | Parameter defines the maximum number of results to return. There are total of four options: 25, 50 (default), 100 and 200 results. walmart | page | string | Value is used to get the items on a specific page. (e.g., 1 (default) is the first page of results, 2 is the 2nd page of results, 3 is the 3rd page of results, etc.). Maximum page value is 100. walmart | ps | number | Determines the number of items per page. There are scenarios where Walmart overrides the ps value. By default Walmart returns 40 results. apple_app_store | num | string | Parameter defines the number of results you want to get per each page. It defaults to 10. Maximum number of results you can get per page is 200. Any number greater than maximum number will default to 200. apple_app_store | page | string | Parameter is used to get the items on a specific page. (e.g., 0 (default) is the first page of results, 1 is the 2nd page of results, 2 is the 3rd page of results, etc.).

Offset + page + size, e.g. yahoo_shopping, home_depot

Engine | Param | Type | Description -- | -- | -- | -- yahoo_shopping | start | number | Parameter defines the result offset. It skips the given number of results. It’s used for pagination. (e.g., 1 (default) is the first page of results, 60 is the 2nd page of results, 120 is the 3rd page of results, etc.). yahoo_shopping | limit | number | Parameter defines the maximum number of results to return. (e.g., 10 (default) returns 10 results, 40 returns 40 results, and 100 returns 100 results). yahoo_shopping | page | string | The page parameter does the start parameter math for you! Just define the page number you want. Pagination starts from 1. home_depot | nao | string | Defines offset for products result. A single page contains 24 products. First page offset is 0, second -> 24, third -> 48 and so on. home_depot | page | string | Value is used to get the items on a specific page. (e.g., 1 (default) is the first page of results, 2 is the 2nd page of results, 3 is the 3rd page of results, etc.). home_depot | ps | number | Determines the number of items per page. There are scenarios where Home depot overrides the ps value. By default Home depot returns 24 results. naver | start | number | Parameter controls the offset of the organic results. This parameter defaults to 1 (except for the web). (e.g. The formula for all searches except the web is start = (page number * 10) - 9 e.g. Page number 3 (3 * 10) - 9 = 21) The formula for the web will be start = (page number * 15) - 29 e.g. Page number 3 (3 * 15) - 29 = 16. naver | num | string | Parameter defines the maximum number of results to return. 50 (default) returns 50 results. Maximum number of results to return is 100.Parameter can only be used with Naver Images API. naver | page | string | The page parameter does the start parameter math for you! Just define the page number you want. Pagination starts from 1.

Token only, e.g. google_scholar_profiles, google_play

Engine | Parameter | Type | Description -- | -- | -- | -- google_scholar_profiles | after_author | string | Parameter defines the next page token. It is used for retrieving the next page results. The parameter has the precedence over before_author parameter. google_scholar_profiles | before_author | string | Parameter defines the previous page token. It is used for retrieving the previous page results. google_maps_photos | next_page_token | string | Parameter defines the next page token. It is used for retrieving the next page results. 20 results are returned per page. google_maps_reviews | next_page_token | string | Parameter defines the next page token. It is used for retrieving the next page results.Usage of start parameter (results offset) has been deprecated by Google. google_play | next_page_token | string | Parameter defines the next page token. It is used for retrieving the next page results.

Possible approaches

The key question is how we might abstract the pagination logic in a manner that makes using SerpApi simpler and more ergonomic.

Approach 1: New function

• New function getPaginatedJson that can be looped over.

  const organicResults = [];
  for await (const page of getPaginatedJson("google", { q: "coffee", start: 15 })) {
  organicResults.push(...page.organic_results);
  if (organicResults.length >= 50) break;
  }

Pros

• Types are clean.
• Iterating over the function to get multiple page results is nice.

Cons

• New function, might be confusing.
• Not very ergonomic since if you want to get the next page, you need to call a different function.
• Does not support callbacks.

Approach 2: Next method

• getJson returns the results object that includes a next method.

  const organicResults = [];
  let page = await getJson("google", { q: "coffee", start: 15 });
  while (page) {
  organicResults.push(...page.organic_results);
  if (organicResults.length >= 50) break;
  page = await page.next();
  }

Pros

• Ergonomic since if you want to get the next page, you can just call the next method on the result object.
• Not a breaking change to existing implementations that use getJson.
• Simpler to understand than using a brand new function.
• Supports callbacks.

Cons

• Cannot iterate over it to get multiple page results.
  • Though technically you can create an async iterable wrapper

Approach 3: Magic?

• getJson returns the results object as per normal.
• If looped over, then it returns each page's results.

  
  // calling once works
  await getJson("google", { q: "coffee", start: 15 });

// calling within a loop works too for await (const page of await getJson("google", { q: "coffee", start: 15 })) { organicResults.push(...page.organic_results); if (organicResults.length >= 50) break; }

### Pros
- Works for single calls or when called in a loop.
- Iterating over the function to get multiple page results is nice.
- Not a breaking change to existing implementations that use `getJson`.
- Simpler to understand than using a brand new function.

### Cons
- Does not support callbacks.
- There are 2 `await`s in the loop, might be confusing.
  - This is required because `getJson` returns a Promise that needs to be awaited to return an object that contains the fetched results and also the instructions necessary to continue the async loop. i.e. returns an [async iterable object](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of)
- Types are a little strange as it includes a `[Symbol.asyncIterator]` key which is required for the loop to work.

[sebastianquek]

Pagination with getJsonBySearchId

getJsonBySearchId effectively returns the same object as getJson . Instead of sending in search params, users provide the search’s ID. It could potentially return the .next() method for pagination.

The flow could be something like this:

1. Get searchId by calling getJson with async=true
2. Call getJsonBySearchId with searchId
3. With the response, call .next() , which in turn calls getJson to get the next page synchronously

This might be useful if a user has 2 different services, one that initiates the request and one that processes + fetches additional requests.

Considerations

• Cannot compare parameters from the function call with the nextParameters extracted from the JSON response
  • The parameters used to compare with nextParameters in this case refers to getJsonBySearchId's parameters, which is not the same as getJson parameters.
  • One way is to extract the parameters from the JSON response. However, we can’t just take search_parameters wholesale as there are some conversions. E.g. for location, the location field doesn’t exist. This might be the same for other parameters too.

    "location_requested":"Austin, Texas, United States",
    "location_used":"Austin,Texas,United States",

• Should .next() trigger an async request?

[alonp123]

In addition to what @sebastianquek has written about the improvements that can be done, I've just realized that the type of num property (in getJson function) is string instead of number.. It would be nice to fix it on the next version..

    /**
     * Result Offset
     * Parameter defines the result offset. It skips the given number of results. It's
     * used for pagination. (e.g., `0` (default) is the first page of results, `10` is
     * the 2nd page of results, `20` is the 3rd page of results, etc.).
     * Google Local Results only accepts multiples of `20`(e.g. `20` for the second
     * page results, `40` for the third page results, etc.) as the start value.
     */
    start?: number;
    /**
     * Number of Results
     * Parameter defines the maximum number of results to return. (e.g., `10` (default)
     * returns 10 results, `40` returns 40 results, and `100` returns 100 results).
     */
    num?: string;