search

singer-io / tap-gitlab

GNU Affero General Public License v3.0
15 stars 19 forks source link

When use api/v4, will only get the last 20 records #21

Open EastGaint opened 3 years ago

EastGaint commented 3 years ago

After check the code, I found the header of api/v4 has not contains "X-Total-Pages", so we cannot get the total page. The "X-Next-Page" is a good solution, if the records are read to end, this attribute will empty.

params = {'page': 1} resp = request(url, params) for row in resp.json(): yield row next_page_s = resp.headers.get('X-Next-Page', 1) while len(next_page_s) != 0: next_page_i = int(next_page_s) params['page'] = next_page_i resp = request(url, params) next_page_s = resp.headers.get('X-Next-Page', 1) for row in resp.json(): yield row

EastGaint commented 3 years ago

def gen_request(url): next_page_s = "1" while len(next_page_s) != 0: next_page_i = int(next_page_s) params = {'page': next_page_i} resp = request(url, params) next_page_s = resp.headers.get('X-Next-Page', 1) for row in resp.json(): yield row

faouzelfassi commented 3 years ago

Hi @EastGaint, thanks for the issue! Is there something new on this one?