Currently in rest_api source, we have a predefined set of paginators to handle various API pagination methods. (E.g. JSONResponsePaginator, HeaderLinkPaginator, SinglePagePaginator, etc.). This works well for built-in paginators, it lacks flexibility for users who wish to introduce custom pagination logic specific to their needs.
Problem
Users must modify the rest_api.config_setup.PAGINATOR_MAP to add new paginators.
Proposal
Introduce a pluggable paginator system that allows users to define and register custom paginators dynamically. This should have a registration function to add new paginators to the existing paginator map and check for duplicates.
Future enhancement can include using entry points for plugin discovery, allowing users to define and load custom paginators from external packages.
Example
from dlt.sources.helpers.rest_client.paginators import BasePaginator
class CustomPaginator(BasePaginator):
...
# Register the custom paginator
register_paginator("custom_paginator", CustomPaginator)
Tasks
[ ] Update the rest_api source to support pluggable paginators.
Background
Currently in
rest_apisource, we have a predefined set of paginators to handle various API pagination methods. (E.g.JSONResponsePaginator,HeaderLinkPaginator,SinglePagePaginator, etc.). This works well for built-in paginators, it lacks flexibility for users who wish to introduce custom pagination logic specific to their needs.Problem
Users must modify the
rest_api.config_setup.PAGINATOR_MAPto add new paginators.Proposal
Introduce a pluggable paginator system that allows users to define and register custom paginators dynamically. This should have a registration function to add new paginators to the existing paginator map and check for duplicates.
Future enhancement can include using entry points for plugin discovery, allowing users to define and load custom paginators from external packages.
Example
Tasks
rest_apisource to support pluggable paginators.