search

kyletaylored / drupal-nodejs-sitemap

Small script for analyzing Drupal content types on sites.
4 stars 1 forks source link

Multiplex http requests #9

Closed MKorostoff closed 5 years ago

MKorostoff commented 5 years ago

Improves scrape time by roughly 8x

kyletaylored commented 5 years ago

So this seems to work better on smaller sites, but larger sites we end up getting timeouts and network issues from fetch. You can see in the recording below that no nodes, forms, or languages are logged, but you can see in the Status Codes the types of errors that were caught.

https://asciinema.org/a/245625

Screen Shot 2019-05-11 at 1 36 17 PM

I added a few catches for errors I was seeing to reduce or handle as many as I can, but we might want to look at using something like Promise Queue instead to limit the number of Promises being executed at once.

Some sitemaps to test: