Enhanced parameter resolution for RESTAPIConfig and REST API source

[ArneDePeuter]

Feature description

Hi!

Currently, in DLT, you can define and resolve parameters dynamically like this:

"endpoint": {
    "path": "issues/{issue_number}/comments",
    "params": {
        "issue_number": {
            "type": "resolve",
            "resource": "issues",
            "field": "number",
        }
    },
},

While this works for some use cases, it's limited in flexibility. Specifically, it does not support dynamic or more generic scenarios, such as resolving parameters for headers or request bodies.

Are you a dlt user?

I'd consider using dlt, but it's lacking a feature I need.

Use case

I need to run data extraction from an API that requires dynamic headers, following these steps:

Authenticate using a login endpoint to retrieve a base token.

• Retrieve a list of items using the base token.
• For each item in the list:
  • Fetch a token specific to that item.
  • Make further requests where the item-specific token is included in the headers.

Example Workflow:

1. POST /authentication -> Returns a base token
2. GET /items -> Uses the base token, returns a list of items
3. For each item:
   • POST /items/{item_id}/token -> Returns an item-specific token
   • GET /items/{item_id}/data -> Uses the item-specific token in headers

While this may reflect poor API design, it is a real-world scenario that requires more flexible parameter resolution capabilities.

Proposed solution

EG for dynamic headers:

"endpoint": {
    "path": "issues/comments",
    "headers": {"token": "{token}"}
    "params": {
        "token": {
            "type": "resolve",
            "resource": "authenticate",
            "field": "token",
        }
    },
},
# or even with static parameters
"endpoint": {
    "path": "issues/comments",
    "headers": {"token": "{token}", "static_param":  region}
    "params": {
        "token": {
            "type": "resolve",
            "resource": "authenticate",
            "field": "token",
        }
    },
},

Related issues

This feature could also hypothetically be extended to support dynamic bodies in the future. While this isn't a current issue for me, it may benefit other use cases where request bodies need to include dynamically resolved fields.

[burnash]

Thanks for the detailed feature request, @ArneDePeuter, and for laying out the use case so clearly! This is a really interesting suggestion. We'll review this further and evaluate how it might be incorporated into rest_api source.

[ArneDePeuter]

Hi @burnash, the PR referenced satisfies the feature. Hope its useful!