voxmedia / tap-instagram

Singer Tap for the Instagram Graph API
Apache License 2.0
5 stars 6 forks source link

handle error in validate_response #12

Closed hkuffel closed 1 year ago

hkuffel commented 1 year ago

Ticket: https://vmproduct.atlassian.net/browse/CDT-441?atlOrigin=eyJpIjoiZGMxZmY2ODMyYWU1NDQxM2FlODA2N2U4NTBmMWRmNzMiLCJwIjoiaiJ9

Solves a problem that was causing premature failure of a bunch of Instagram pipelines.

Before, if a number of different streams encountered a 400 error (which is quite common and not necessarily a big problem), the entire meltano job would fail.

Now, if any stream that inherits from the parent InstagramStream object encounters that specific response (a 400 error with "Unsupported get request" in the message), the tap will log that it's skipping that stream instance and moving on to the next one.