Added request tracing and API pagination support

[justinpolygon]

This PR introduces improvements around request tracing and pagination, although distinct, I chose to bundle them into a single PR due to their interrelatedness. When dealing with APIs returning paginated results, figuring out what urls are being fetched becomes pretty complex without added logging. So, just adding the request tracing feature alongside pagination support, greatly simplifies the debugging -- that's why you see the two improvements in one PR. We added tracing to the python client already and will do the same for Go. So, this adds consistency too.

Request Tracing: A new optional trace parameter in globalOptions enables request and response logging for debugging purposes. When activated, the request URL, sanitized headers, and response headers are logged in the console.

Here's an example script:

import('@polygon.io/client-js').then(({ restClient }) => {
    const globalFetchOptions = {
        trace: true
    };
    const rest = restClient("XXXXX", "https://api.polygon.io", globalFetchOptions);

    rest.stocks.aggregates("AAPL", 1, "minute", "2019-01-01", "2023-02-01", { limit: 50000 }).then((data) => {
        //  console.log(data);
        // Count the results
        const resultCount = data.length;
        console.log("Result count:", resultCount);
    }).catch(e => {
        console.error('An error happened:', e);
    });
});

The output looks like this:

root@b3d6fa7e1b1a:~/polygon/client-js-testing# node examples/rest/stocks-aggregates_bars.js
Request URL:  https://api.polygon.io/v2/aggs/ticker/AAPL/range/1/minute/2019-01-01/2023-02-01?limit=50000
Request Headers:  { Authorization: 'Bearer REDACTED' }
Response Headers:  Headers {
  [Symbol(map)]: [Object: null prototype] {
    server: [ 'nginx/1.19.2' ],
    date: [ 'Thu, 15 Jun 2023 06:32:32 GMT' ],
    'content-type': [ 'application/json' ],
    'transfer-encoding': [ 'chunked' ],
    connection: [ 'close' ],
    'content-encoding': [ 'gzip' ],
    vary: [ 'Accept-Encoding' ],
    'x-request-id': [ '51e0ec5ce99715f726192787a960c7ac' ],
    'strict-transport-security': [ 'max-age=15724800; includeSubDomains' ]
  }
}

Which can be extremely useful for figuring out what url API you're talking to and what the client and server headers are (the x-request-id is really useful for support).

Pagination Support: getWithGlobals can now handle APIs returning paginated results (fixes https://github.com/polygon-io/client-js/issues/103). The function will automatically fetch all pages of data when the API response indicates more data is available.

In the old version, each call to getWithGlobals could only retrieve a single "page" of data from the server. This would only provide the first page of results, and the end user application would need to have logic to retrieve all of the data.

Here's a test script (current version):

import('@polygon.io/client-js').then(({ restClient }) => {
    const rest = restClient("XXXXX", "https://api.polygon.io");

    rest.stocks.aggregates("AAPL", 1, "minute", "2019-01-01", "2023-02-01", { limit: 50000 }).then((data) => {
        //  console.log(data);
        // Count the results
        const resultCount = data.length;
        console.log("Result count:", resultCount);
    }).catch(e => {
        console.error('An error happened:', e);
    });
});

Here's the output:

root@b3d6fa7e1b1a:~/polygon/client-js-testing# node examples/rest/stocks-aggregates_bars.js
Result count: 50000

In the new version, the getWithGlobals function uses a nested function fetchPage to fetch a page of data, and then check if there's another page to fetch. This is done by looking for a next_url property in the JSON response. If there is a next_url, it calls itself recursively to fetch the next page, passing along the accumulated data so far.

Here's a test script (new version -- try turning on tracing too):

import('@polygon.io/client-js').then(({ restClient }) => {
    const globalFetchOptions = {
        trace: true
    };
    const rest = restClient("XXXXX", "https://api.polygon.io", globalFetchOptions);

    rest.stocks.aggregates("AAPL", 1, "minute", "2019-01-01", "2023-02-01", { limit: 50000 }).then((data) => {
        //  console.log(data);
        // Count the results
        const resultCount = data.length;
        console.log("Result count:", resultCount);
    }).catch(e => {
        console.error('An error happened:', e);
    });
});

Here's the output:

root@b3d6fa7e1b1a:~/polygon/client-js-testing# node examples/rest/stocks-aggregates_bars.js
Result count: 796098

The best part is, these changes are entirely transparent to the user and require no changes to their existing code. This hopefully provides a "just works" experience for users of the client-js library.

[justinpolygon]

Hey @timetraveler328, I had one though last night, I think the only risk here is that it's so seamless that it could potentially break existing users if they have rolled their own pagination logic. So, we'll likely need to cut a new major release and then add a breaking change notice or something along those lines.

[nmatkins]

Hey @timetraveler328, I had one though last night, I think the only risk here is that it's so seamless that it could potentially break existing users if they have rolled their own pagination logic. So, we'll likely need to cut a new major release and then add a breaking change notice or something along those lines.

For now could you make it not automatically paginate, but let users opt in with like a pagination: true parameter?

This would be a pretty useful feature, otherwise it is clunky getting all the possible results.

[justinpolygon]

Hey @nmatkins, thanks for the suggestion. This makes sense to me. I've added a pagination flag for turning on and off fetching the next_url if there is one. The default is off.

Here's an example script:

import('@polygon.io/client-js').then(({ restClient }) => {
    const globalFetchOptions = {
        trace: true,
        pagination: true,
    };
    const rest = restClient("XXXX", "https://api.polygon.io", globalFetchOptions);

    rest.stocks.aggregates("AAPL", 1, "minute", "2019-01-01", "2023-02-01", { limit: 50000 }).then((data) => {
        //  console.log(data);
        // Count the results
        const resultCount = data.length;
        console.log("Result count:", resultCount);
    }).catch(e => {
        console.error('An error happened:', e);
    });
});

Here's the output with pagination set to false:

$ node examples/rest/t.js
Request URL:  https://api.polygon.io/v2/aggs/ticker/AAPL/range/1/minute/2019-01-01/2023-02-01?limit=50000
Request Headers:  { Authorization: 'Bearer REDACTED' }
Response Headers:  Headers {
  [Symbol(map)]: [Object: null prototype] {
    server: [ 'nginx/1.19.2' ],
    date: [ 'Thu, 06 Jul 2023 18:34:27 GMT' ],
    'content-type': [ 'application/json' ],
    'transfer-encoding': [ 'chunked' ],
    connection: [ 'close' ],
    'content-encoding': [ 'gzip' ],
    vary: [ 'Accept-Encoding' ],
    'x-request-id': [ '06dc97920681d8335c0451894aa1f79f' ],
    'strict-transport-security': [ 'max-age=15724800; includeSubDomains' ]
  }
}
Result count: 50000

Here's the output with pagination set to true:

$ node examples/rest/t.js
Request URL:  https://api.polygon.io/v2/aggs/ticker/AAPL/range/1/minute/2019-01-01/2023-02-01?limit=50000
Request Headers:  { Authorization: 'Bearer REDACTED' }
Response Headers:  Headers {
  [Symbol(map)]: [Object: null prototype] {
    server: [ 'nginx/1.19.2' ],
    date: [ 'Thu, 06 Jul 2023 18:19:29 GMT' ],
    'content-type': [ 'application/json' ],
    'transfer-encoding': [ 'chunked' ],
    connection: [ 'close' ],
    'content-encoding': [ 'gzip' ],
    vary: [ 'Accept-Encoding' ],
    'x-request-id': [ 'b3458569be9ad6d30a3c8b1cc8e2035f' ],
    'strict-transport-security': [ 'max-age=15724800; includeSubDomains' ]
  }
}
...

Request URL:  https://api.polygon.io/v2/aggs/ticker/AAPL/range/1/minute/1667855220000/2023-02-01?cursor=bGltaXQ9NTAwMDAmc29ydD1hc2M
Request Headers:  { Authorization: 'Bearer REDACTED' }
Response Headers:  Headers {
  [Symbol(map)]: [Object: null prototype] {
    server: [ 'nginx/1.19.2' ],
    date: [ 'Thu, 06 Jul 2023 18:19:47 GMT' ],
    'content-type': [ 'application/json' ],
    'transfer-encoding': [ 'chunked' ],
    connection: [ 'close' ],
    'content-encoding': [ 'gzip' ],
    vary: [ 'Accept-Encoding' ],
    'x-request-id': [ '60c9a1545c0d717fbddd98c188febcf1' ],
    'strict-transport-security': [ 'max-age=15724800; includeSubDomains' ]
  }
}
Result count: 796098

[justinpolygon]

FYI - @timetraveler328 think we're ready for a review now. Thx.

[nmatkins]

Hey @nmatkins, thanks for the suggestion. This makes sense to me. I've added a pagination flag for turning on and off fetching the next_url if there is one. The default is off.

Here's an example script:

import('@polygon.io/client-js').then(({ restClient }) => {
  const globalFetchOptions = {
      trace: true,
      pagination: true,
  };
  const rest = restClient("XXXX", "https://api.polygon.io", globalFetchOptions);

  rest.stocks.aggregates("AAPL", 1, "minute", "2019-01-01", "2023-02-01", { limit: 50000 }).then((data) => {
      //  console.log(data);
      // Count the results
      const resultCount = data.length;
      console.log("Result count:", resultCount);
  }).catch(e => {
      console.error('An error happened:', e);
  });
});

Here's the output with pagination set to false:

$ node examples/rest/t.js
Request URL:  https://api.polygon.io/v2/aggs/ticker/AAPL/range/1/minute/2019-01-01/2023-02-01?limit=50000
Request Headers:  { Authorization: 'Bearer REDACTED' }
Response Headers:  Headers {
  [Symbol(map)]: [Object: null prototype] {
    server: [ 'nginx/1.19.2' ],
    date: [ 'Thu, 06 Jul 2023 18:34:27 GMT' ],
    'content-type': [ 'application/json' ],
    'transfer-encoding': [ 'chunked' ],
    connection: [ 'close' ],
    'content-encoding': [ 'gzip' ],
    vary: [ 'Accept-Encoding' ],
    'x-request-id': [ '06dc97920681d8335c0451894aa1f79f' ],
    'strict-transport-security': [ 'max-age=15724800; includeSubDomains' ]
  }
}
Result count: 50000

Here's the output with pagination set to true:

$ node examples/rest/t.js
Request URL:  https://api.polygon.io/v2/aggs/ticker/AAPL/range/1/minute/2019-01-01/2023-02-01?limit=50000
Request Headers:  { Authorization: 'Bearer REDACTED' }
Response Headers:  Headers {
  [Symbol(map)]: [Object: null prototype] {
    server: [ 'nginx/1.19.2' ],
    date: [ 'Thu, 06 Jul 2023 18:19:29 GMT' ],
    'content-type': [ 'application/json' ],
    'transfer-encoding': [ 'chunked' ],
    connection: [ 'close' ],
    'content-encoding': [ 'gzip' ],
    vary: [ 'Accept-Encoding' ],
    'x-request-id': [ 'b3458569be9ad6d30a3c8b1cc8e2035f' ],
    'strict-transport-security': [ 'max-age=15724800; includeSubDomains' ]
  }
}
...

Request URL:  https://api.polygon.io/v2/aggs/ticker/AAPL/range/1/minute/1667855220000/2023-02-01?cursor=bGltaXQ9NTAwMDAmc29ydD1hc2M
Request Headers:  { Authorization: 'Bearer REDACTED' }
Response Headers:  Headers {
  [Symbol(map)]: [Object: null prototype] {
    server: [ 'nginx/1.19.2' ],
    date: [ 'Thu, 06 Jul 2023 18:19:47 GMT' ],
    'content-type': [ 'application/json' ],
    'transfer-encoding': [ 'chunked' ],
    connection: [ 'close' ],
    'content-encoding': [ 'gzip' ],
    vary: [ 'Accept-Encoding' ],
    'x-request-id': [ '60c9a1545c0d717fbddd98c188febcf1' ],
    'strict-transport-security': [ 'max-age=15724800; includeSubDomains' ]
  }
}
Result count: 796098

Thanks Justin!

[justinpolygon]

LGTM - @mpatel18 I was chatting with Katie and it sounds like this needs to be merged into the release branch. I'm not sure of the release procedure here. If we merge this can you also merge https://github.com/polygon-io/client-js/pull/176? These are just the docs for these two changes.

[mpatel18]

LGTM - @mpatel18 I was chatting with Katie and it sounds like this needs to be merged into the release branch. I'm not sure of the release procedure here. If we merge this can you also merge #176? These are just the docs for these two changes.

Gotcha @justinpolygon, if you could get this and #176 merged I can handle the release or would it fine for me to merge these. I think its fine as long as both go into master, we have a different release branch pattern for version bumps.

[justinpolygon]

Okay, @mpatel18 both are merged into master. Thanks for all your help here!