dlt-hub / verified-sources

Contribute to dlt verified sources 🔥
https://dlthub.com/docs/walkthroughs/add-a-verified-source
Apache License 2.0
48 stars 38 forks source link

rest_api: accept `DltResource` instances in resource list #473

Open rudolfix opened 1 month ago

rudolfix commented 1 month ago

Source name

rest_api

Describe the data you'd like to see

We could combine instances of regular resources with a. Such config could look like this:

@dlt.resource(name="stories")
def read_stories(limit: int = 10):
    ...

@dlt.source(name="hacker_news_source", max_table_nesting=2)
def hacker_news_source(
    base_url: str = dlt.config.value,
) -> List[DltResource]:

    # source configuration
    source_config: RESTAPIConfig = {
        "client": {
            "base_url": base_url,
        },
        "resources": [
            {
                "name": "item",
                "table_name": "item",
                "endpoint": {
                    "path": "/item/{id}.json",
                    "data_selector": "$",
                    "paginator": "single_page",
                    "params": {
                        "id": {
                            "type": "resolve",
                            "field": "$",
                            "resource": "stories"
                        }
                    },
                },
            },
            read_stories(20)
        ],
    }

    return rest_api_source(source_config)

Above we add resource instance with the name "stories" and also attach it to a transformer that is fully declared

Are you a dlt user?

Yes, I'm already a dlt user.

Do you ready to contribute this extension?

Yes, I'm ready.

dlt destination

No response

Additional information

No response

burnash commented 1 month ago

TODO: Update the docs after this feature is merged