method 1 (query parameter level) has less features than method 2 (resource level) because method 1 does not support the end_param. The reason is that it is nested as a child of the start_param.
The code for both methods is redundant
users might be confused by having two APIs for the same thing but one API being slightly less powerful
the rest_api source creates a dlt.sources.Incremental. However, the current integration with that incremental class might not be ideal because the rest_api source holds and applies the transform function, which allows value transformations, such as epoch to datetime.
Proposal
Provide only one way to specify the incremental loads or make both ways equally powerful
Move the transformation function into the dlt.sources.Incremental class and make it a method. Users can then pass either an instance of the dlt.sources.Incremental or a dictionary.
Source name
rest_api
Describe the data you'd like to see
Currently, the declarative rest_api source offers two APIs to specify incremental data loads:
Example for method 1:
Example for method 2:
This poses some challenges:
end_param
. The reason is that it is nested as a child of thestart_param
.dlt.sources.Incremental
. However, the current integration with that incremental class might not be ideal because the rest_api source holds and applies thetransform
function, which allows value transformations, such as epoch to datetime.Proposal
dlt.sources.Incremental
class and make it a method. Users can then pass either an instance of thedlt.sources.Incremental
or a dictionary.Are you a dlt user?
Yes, I'm already a dlt user.
Do you ready to contribute this extension?
Yes, I'm ready.