Archive API Requests Returning Partial Responses

mattmakesmaps commented 7 years ago

Hello, and happy 4th of July 🎆🇺🇸🎆

I'm hacking on the Archive API, and am running into some behavior I wasn't expecting.

In my node script, I send two requests for two different month's worth of data, three seconds apart. I'm using the request-promise library to issue the requests. Sometimes these will complete successfully, but often they will not.

Specifically, the response will come back with a 200 status code, but the JSON body will be incomplete.

I was able to replicate this behavior using Charles, making back-to-back queries for February 1985, over 30 seconds apart.

Here are gists representing the first complete response, and the second partial response.

My main questions are:

Is there a problem with the format of the requests I'm sending to the API?
Is there a best practice I should follow to both format the requests and process the response, so that I could either stop getting partial responses or limit the chances that a partial response would be returned?

Thanks and please let me know if there are additional details I can provide.

nyt-hughmandeville commented 7 years ago

The responses from the Archive API can be pretty big (~20mb). Does your request have a timeout?

The Archive API is meant so you can build up your own database locally of NYT article metadata.

https://developer.nytimes.com/archive_api.json#/README

mattmakesmaps commented 7 years ago

Hi and thanks for the quick response. My request doesn't have a timeout associated with it.

My workaround was basically to check if the response was a valid JSON object, and retry if not. Seems to be working fine.

nytimes / public_api_specs

Archive API Requests Returning Partial Responses #30