search

ajhyndman / fire-emblem-working-title

A fledgling project wrangling Fire Emblem: Heroes stats
25 stars 6 forks source link

Introduce the `fetchApiRows` function for extracting particularly large Cargo table result sets #128

Closed carsomyr closed 6 years ago

carsomyr commented 6 years ago

With the recent banner, the number of passive skills has exceeded an invisible 500 Cargo row threshold, resulting in a truncated list of skills. You can see how skills whose names begin with "W", like "Wings of Mercy 1", are missing after running npm run scrape-stats.

This commit introduces the fetchApiRows function, which scraps the API_LIMIT constant and instead fetches from Cargo tables with a batch size of 500, incrementing the offset query parameter by API_BATCH_SIZE and querying again after each partial result set.

ajhyndman commented 6 years ago

Just once, I want two consecutive scrapes to work without requiring changes! 😭

LGTM 👍