search

meltano / sdk

Write 70% less code by using the SDK to build custom extractors and loaders that adhere to the Singer standard: https://sdk.meltano.com
https://sdk.meltano.com
Apache License 2.0
94 stars 68 forks source link

Streamline complementary REST requests #93

Open MeltyBot opened 3 years ago

MeltyBot commented 3 years ago

Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/93

Originally created by @aaronsteers on 2021-04-08 23:56:48

There are at least two use cases for developers to make additional REST API calls beyond what is handled already by the SDK:

  1. During discovery - The tap may want to ping a REST API ahead of calls to get_records().
  2. During data-append - The tap may need to supliment the core data with additional calls in the post_process() handler.

For both of these cases, we should establish a pattern that developers can implement with confidence.

As of today, the developer would likely implement calls against the requests library directly. The biggest downside of this approach is that those extra calls may have inconsistencies in regards to authentication and retry logic.

MeltyBot commented 2 years ago

View 5 previous comments from the original issue on GitLab

stale[bot] commented 1 year ago

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

edgarrmondragon commented 1 year ago

Could be addressed by #1649

ReubenFrankel commented 4 months ago

Curious if there is any more progress on this?

edgarrmondragon commented 4 months ago

Curious if there is any more progress on this?

Not really 😕. Did you have a particular use case is mind?

ReubenFrankel commented 4 months ago

Curious if there is any more progress on this?

Not really 😕. Did you have a particular use case is mind?

Definitely a nice-to-have rather than a pressing issue, but same as my comment on the GitLab issue - tap-auth0 has some clunky logic to request a user export job, then poll the job until it's completed in order to fetch the resulting gzip, decompress and parse out records.

https://github.com/Matatika/tap-auth0/blob/75d5c92d57afe3443f793a3f5a671cb6da826af8/tap_auth0/streams.py#L22-L117

edgarrmondragon commented 3 months ago

Curious if there is any more progress on this?

Not really 😕. Did you have a particular use case is mind?

Definitely a nice-to-have rather than a pressing issue, but same as my comment on the GitLab issue - tap-auth0 has some clunky logic to request a user export job, then poll the job until it's completed in order to fetch the resulting gzip, decompress and parse out records.

https://github.com/Matatika/tap-auth0/blob/75d5c92d57afe3443f793a3f5a671cb6da826af8/tap_auth0/streams.py#L22-L117

I'll revisit #1649 and see if I can a) make sure the API in that PR would truly solve this and b) get it to a merge-able state

ReubenFrankel commented 3 months ago

Appreciate it - no rush though. 😄