rest_api: pluggable paginators

[burnash]

Background

Currently in rest_api source, we have a predefined set of paginators to handle various API pagination methods. (E.g. JSONResponsePaginator, HeaderLinkPaginator, SinglePagePaginator, etc.). This works well for built-in paginators, it lacks flexibility for users who wish to introduce custom pagination logic specific to their needs.

Problem

Users must modify the rest_api.config_setup.PAGINATOR_MAP to add new paginators.

Proposal

Introduce a pluggable paginator system that allows users to define and register custom paginators dynamically. This should have a registration function to add new paginators to the existing paginator map and check for duplicates.

Future enhancement can include using entry points for plugin discovery, allowing users to define and load custom paginators from external packages.

Example

from dlt.sources.helpers.rest_client.paginators import BasePaginator

class CustomPaginator(BasePaginator):
    ...

# Register the custom paginator
register_paginator("custom_paginator", CustomPaginator)

Tasks

• [x] Update the rest_api source to support pluggable paginators.
• [x] Add tests.
• [x] Update the docs in the dlt repo.

[willi-mueller]

The same holds true for authenticators.