search

The-Law-1 / HOTSHelper

Compiles information from hotslogs to help you build the best team in Blizzard's Heroes Of The Storm
0 stars 0 forks source link

Optimize scraping #13

Open The-Law-1 opened 2 years ago

The-Law-1 commented 2 years ago

Scrape all the rows at once as an array and flip through that instead of scraping in the while loop

The-Law-1 commented 2 years ago

Use browser.wsEndpoint() and puppeteer.connect to save some time

https://stackoverflow.com/questions/69170403/puppeteer-using-only-1-browser-instance Basically only one browser, I don't know would you just never close it or what, maybe a text file with the current endpoint ?

The-Law-1 commented 2 years ago

It turns out the browser opens fast, but it's the page that takes time to load

The-Law-1 commented 2 years ago

page metrics: https://github.com/puppeteer/puppeteer/blob/main/docs/api.md#pagemetrics

The-Law-1 commented 2 years ago

The cache helps a little (1-2 seconds)