Groupsio currently has a bug where the api for gettopics is not returning correct pagination values hence causing our stream worker to error out. Here's an overview of the bug
the start_item and end_item of each page always stays 1 and 10 for which ever page it is returning when the expectation is it should return the current page item eg, in a page size of 10 page 1 should be start_item=1 and end_item=10, page 2 should be start_item=11 and end_item=20
Even when you are on the last page has_more always returns true hence can't tell from the payload if we are on the last page and hence make a request for a next page that doesn't exist.
This PR is a work around on the issues above by
Managing the pagination in code instead of relying on the api by using the total count of the payload and sending extra params to each stream to indicate the page we are on.
This also calculates if the current page is the last to avoid calling a next page that doesn't exist and hence erroring out.
Changes proposed ✍️
What
Groupsio currently has a bug where the api for
gettopicsis not returning correct pagination values hence causing our stream worker to error out. Here's an overview of the bugstart_itemandend_itemof each page always stays 1 and 10 for which ever page it is returning when the expectation is it should return the current page item eg, in a page size of 10 page 1 should bestart_item=1andend_item=10, page 2 should bestart_item=11andend_item=20has_morealways returns true hence can't tell from the payload if we are on the last page and hence make a request for a next page that doesn't exist.This PR is a work around on the issues above by
total countof the payload and sending extra params to each stream to indicate the page we are on.