search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.19k stars 4.14k forks source link

Auth0 pagination is not handling more than 1 page of results #29358

Open vasilisgav opened 1 year ago

vasilisgav commented 1 year ago

Connector Name

source-auth0

Connector Version

0.3.0

What step the error happened?

During the sync

Revelant information

The paginator in source-auth0 expects in body the length property https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-auth0/source_auth0/source.py#L47

but the body has no such field. so it always lands here https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-auth0/source_auth0/source.py#L65-L71

as result only 50 results at a time are retrieved.

To produce this, pick a source that with a single request retrieves more than 50 results, its easier with streams such as: clients, users, organizations

Relevant log output

No response

Contribute

vasilisgav commented 1 year ago

@sajarin can you please reopen this? this is not closed

sajarin commented 1 year ago

@vasilisgav can you confirm this isn't working on version 0.5.1 of the connector?

vasilisgav commented 1 year ago

The issue is even bigger, it cannot even be tested, the connection is now broken because of this https://github.com/airbytehq/airbyte/issues/29789