search

TaruDesigns / turinsights_backend

0 stars 0 forks source link

Pagination for data ingestion #7

Open TaruDesigns opened 2 months ago

TaruDesigns commented 2 months ago

Especially on first run, pagination for data ingestion is required otherwise each refresh takes way too long and blocks the worker from other tasks.

TaruDesigns commented 2 months ago

Getting first a "count" isn't worth it performancewise. The remote server still has to fetch everything.