dlt-hub / verified-sources

Contribute to dlt verified sources 🔥
https://dlthub.com/docs/walkthroughs/add-a-verified-source
Apache License 2.0
48 stars 38 forks source link

rest_api: Allow the REST API config object to exclude rows, columns, and transform data #494

Open francescomucio opened 2 weeks ago

francescomucio commented 2 weeks ago

Source name

rest_api

Describe the data you'd like to see

I would like to have the possibility to configure a source like:

            {
                "name": "my_nicely_named_resource",
                "endpoint": {
                    "path": "endpoint_name",
                },
                "row_filter": lambda x: x["id"] == 3,
                "exclude_columns": ["$.id", "$.another_column"],
                "transform": my_function,
            },

Are you a dlt user?

Yes, I'm already a dlt user.

Do you ready to contribute this extension?

Yes, I'm ready.

dlt destination

duckdb, snowflake

Additional information

No response