search

janreges / siteone-crawler

SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).
https://crawler.siteone.io/
MIT License
255 stars 17 forks source link

New option for the OfflineWebsiteExporter #16

Closed mammo0 closed 2 months ago

mammo0 commented 2 months ago

Hello,

I came over an issue during export of a website that has images with very long file names. Because of the length they could not be saved on my filesystem. Additionally the whole offline export stopped because of the following thrown exception: https://github.com/janreges/siteone-crawler/blob/9798252901dd25797d1d38fa26a19c6dbc409fa1/src/Crawler/Export/OfflineWebsiteExporter.php#L169

This exception quits the for loop: https://github.com/janreges/siteone-crawler/blob/9798252901dd25797d1d38fa26a19c6dbc409fa1/src/Crawler/Export/OfflineWebsiteExporter.php#L84-L93

Therefore, I added an option to OfflineWebsiteExporter that ignores such errors: --ignore-store-file-error If it is set, the above exception won't be thrown.

janreges commented 2 months ago

Thank you @mammo0 for this useful option.