Concurrently crawling

[palmiak]

This is the implementation of the discussion https://github.com/leonstafford/wp2static/discussions/762

It would be great if you could take a second look.

There are some things I'm having problems with:

• there are three places that throw a warning about too long lines
• I missed this part, so this is something that should be re-add

  try {
          $response = $this->client->send( $request );
      } catch ( TooManyRedirectsException $e ) {
          WsLog::l( "Too many redirects from $url" );
      }

• also crawlURL() method is no longer used - so we should either remove it or deprecate

Currently the concurrency rate can be set with apply_filters( 'wp2static_concurrent_crawl_rate', 1 ) and is set to 1 as default.

I'm using the new crawler on production for WordPressowka.pl and wpowls.co.

[john-shaffer]

Excellent work!

I resolved the issues you mentioned and added a crawlConcurrency option on the advanced options page. I changed the filter name to wp2static_crawl_concurrency.

On some very small VMs (t3a.micro), I'm seeing improvements from ~10.5s to ~7.5s on a small site with crawlConcurrency=64, and from ~8m to ~6m on a larger slow site with crawlConcurrency=8. Higher concurrency values are slightly slower, but not by much.

[palmiak]

This is really an amazing improvement. So it looks like this 25-30% is a quite repeatable thing.

sob., 6 lis 2021, 02:46 użytkownik John Shaffer @.***> napisał:

Excellent work!

I resolved the issues you mentioned and added a crawlConcurrency option on the advanced options page. I changed the filter name to wp2static_crawl_concurrency.

On some very small VMs (t3a.micro), I'm seeing improvements from ~10.5s to ~7.5s on a small site with crawlConcurrency=64, and from ~8m to ~6m on a larger slow site with crawlConcurrency=8.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/leonstafford/wp2static/pull/834#issuecomment-962361500, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAR34OQVIV46WFQXG3AQF5TUKSJJVANCNFSM5HJQN7DQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.