search

vezaynk / Sitemap-Generator-Crawler

PHP script to recursively crawl websites and generate a sitemap. Zero dependencies.
https://www.bbss.dev
MIT License
243 stars 93 forks source link

change way scanned urls are stored #56

Closed jandanielcz closed 7 years ago

jandanielcz commented 7 years ago

should be faster for large sites. For my testsite is it not visible, but theoretically should be better: http://maettig.com/1397246220

vezaynk commented 7 years ago

I ran a benchmark against http://blog.webfwd.org and literally no difference.

The website was 573 pages and it took 93.3±1 seconds with both scripts. Incidentally, your patch was slower by 2 seconds.

According to the linked website, the expectation would have been along the lines of 15 second speed increase. I am tempted to assume that PHP 7 brought some very nice performance improvements in that domain.

vezaynk commented 7 years ago

If the performance gain is non-existant and adds a layer of confusion to the code, this shouldn't be merged. If you are passionate about this, try to come up with some reproducible benchmarks.

jandanielcz commented 7 years ago

ok, no problem.