dlt-hub / dlt

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
https://dlthub.com/docs
Apache License 2.0
2.65k stars 176 forks source link

[cli] add explicit paginators to rest_api template #1904

Closed AstrakhantsevaAA closed 1 month ago

AstrakhantsevaAA commented 1 month ago

users have some issues with dlt init rest_api duckdb command, I set paginators explicitly, because they are not detected automatically for some reason. also reduced the amount of data for github api (request data for a week, not for month) because of rate limits

netlify[bot] commented 1 month ago

Deploy Preview for dlt-hub-docs canceled.

Name Link
Latest commit 559f344d61ee53f63679d0cef9ae45fc892a5273
Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/66fbeecc09c7a20008b93b35
burnash commented 1 month ago

I set paginators explicitly, because they are not detected automatically for some reason

Can you reproduce it? If yes, let's create an issue for this and fix it. (but we still can explicitly define the paginators of course)

burnash commented 1 month ago

If we want to explicitly define pagination, we'd need to update the tutoral as it says that we didn't explicitly specify pagination: https://dlthub.com/docs/tutorial/rest-api#pagination

AstrakhantsevaAA commented 1 month ago

@burnash

Can you reproduce it? If yes, let's create an issue for this and fix it.

Yes, I can, unfortunately, for both sources :( Let's create an issue then.

Screenshot 2024-10-01 at 15 48 57

burnash commented 1 month ago

Yes, I can, unfortunately, for both sources :( Let's create an issue then.

@AstrakhantsevaAA I think it's not an actual error, but a not very good warning message. Most likely we need to change the message. This Fallback pagiantor warning comes from the child (dependent) resource and informs that there's no paginator on this page.

AstrakhantsevaAA commented 1 month ago

@burnash yes, I see it is just a warning, but also it is the cause of the rate limit error. Not sure why it is happening, but if i add pagination explicitly then the data loads without any problem and rate limit errors. It seems like it keeps requesting one page until the rate limit error, so you can see the list of these warnings (with pagination it takes 1-2 requests).

created the issue: https://github.com/dlt-hub/dlt/issues/1915