crwlrsoft / crawler

Library for Rapid (Web) Crawler and Scraper Development
https://www.crwlr.software/packages/crawler
MIT License
312 stars 11 forks source link

Fix reading uncompressed files #128

Closed otsch closed 7 months ago

otsch commented 7 months ago

When FileCache compression is activated, the FileCache assumed that all cache files that it will read are compressed. But there could be existing cache files created without compression activated which lead to errors. The FileCache now first checks if the content is compressed and only tries to decode if it is.