search

ArchiveTeam / grab-site

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Other
1.32k stars 130 forks source link

Benchmark cpython 3.4 vs Nuitka-built grab-site.exe #128

Closed ivan closed 5 years ago

ivan commented 5 years ago

and add Nuitka-based install instructions if there is any noticeable reduction in CPU time for crawling a site.

ivan commented 5 years ago

I couldn't see any performance difference in short tests, and the Nuitka-built grab-site.exe exhibits multi-gigabyte memory leaks after a while, so closing.