Hi we're utilizing a helper class to parse over the paginated lists as so:
from typing import Any, Callable, ParamSpec, TypeVar
from merge.client import Merge
from merge.core import ApiError
from merge.resources.hris import Employee, Group
_T = ParamSpec("_T")
_R = TypeVar("_R", covariant=True)
class Middleware:
def __init__(self, merge_client: Merge):
self.merge_client = merge_client
def _executor(self, func: Callable[_T, _R], *args: _T.args, **kwargs: _T.kwargs) -> list[Any]:
"""Exhausts a paginated API until all data is retrieved"""
data = []
cursor = ""
while cursor is not None:
try:
next_page = func(*args, **kwargs, cursor=cursor) # type: ignore[arg-type]
for result in next_page.results: # type: ignore[attr-defined]
data.append(result)
cursor = next_page.next # type: ignore[attr-defined]
except Exception as e:
logger.error(
f"Error while fetching users from Merge. Breaking pagination early: {e}"
)
break
return data
def get_users(self) -> list[Employee]:
return self._executor(self.merge_client.hris.employees.list)
Noticing the mypy type ignores, we're having issues with the ParamSpec variable and the return type generic TypeVar covariant. It would be great if all return list return types could subclass from a base Pydantic model as all paginated pydantic models shown are like:
class PaginatedEmployeeList(pydantic.BaseModel):
next: typing.Optional[str]
previous: typing.Optional[str]
results: typing.Optional[typing.List[Employee]]
where the only difference is the "Employee" model which could easily be made a template variable. This way, we can make the return type the base pagination list Pydantic model instead of a generic TypeVar and remove the type ignores.
Hi we're utilizing a helper class to parse over the paginated lists as so:
Noticing the mypy type ignores, we're having issues with the ParamSpec variable and the return type generic TypeVar covariant. It would be great if all return list return types could subclass from a base Pydantic model as all paginated pydantic models shown are like:
where the only difference is the "Employee" model which could easily be made a template variable. This way, we can make the return type the base pagination list Pydantic model instead of a generic TypeVar and remove the type ignores.
Thanks:)