FatalAPIError: 400 Client Error when offset parameter exceeds 9k

[MarcosPala]

Hi,

I encounter a singer_sdk.exceptions.FatalAPIError: 400 Client Error when I am trying to extract +40k results of payment information. After playing around a bit with Postman I discovered that the offset's parameter of the payments/search endpoint has a limitation and It cannot exceed 9000. So when the extraction surpasses this value you get a 400 "bad_request" & "Offset parameter is too large".

I am attaching some images to show you what I mean the log with the 400 Client Error.

Any ideas on how to circumvent this limitation?

elt.log

[DouweM]

@MarcosPala On https://es.stackoverflow.com/questions/227525/api-mercadopago-obtener-json-sin-l%C3%ADmites, a user wrote "Solo se pueden obtener los pagos de los últimos 8 meses." ("You can only obtain payments from the past 8 months") Could the issue you're be not about the exact offset, but about how old those payments are? You could verify this by looking at the date of the last payment that did properly get extracted.

If this is the case, the tap should respond to error 400 with message "Offset parameter is too large" not by failing hard by by considering this stream done and printing a log message saying it can't extract any older records.

[edgarrmondragon]

I've proposed a new exception that would let developers gracefully finalize a stream or partition and continue to the next one: https://github.com/meltano/sdk/issues/280#issuecomment-1237645534 🙂.

[MarcosPala]

@MarcosPala On https://es.stackoverflow.com/questions/227525/api-mercadopago-obtener-json-sin-l%C3%ADmites, a user wrote "Solo se pueden obtener los pagos de los últimos 8 meses." ("You can only obtain payments from the past 8 months") Could the issue you're be not about the exact offset, but about how old those payments are? You could verify this by looking at the date of the last payment that did properly get extracted.

If this is the case, the tap should respond to error 400 with message "Offset parameter is too large" not by failing hard by by considering this stream done and printing a log message saying it can't extract any older records.

Hey Douwe, thanks for answering.

I just found the time to look into this again. I looked into the data I was extracting and I have some a payment from 2021-01-26 19:11:16 UTC that I was able to extract, so I don't think there is an 8 month limitation.

The error that is showing, as you can read it on the log, is a FatalAPIError: 400 Client Error: Bad Request for path due to using an offset bigger than 9000

[DouweM]

@MarcosPala Good catch, seems like they indeed have an arbitrary limit at offset=9000. Looks like @edgarrmondragon has an idea on how to let the SDK handle this better. Would that be sufficient for you, or do you actual need the older data? Doesn't seem like there's anything we can do if this is an API restriction, unfortunately.