search

scrapy-plugins / scrapy-splash

Scrapy+Splash for JavaScript integration
BSD 3-Clause "New" or "Revised" License
3.14k stars 451 forks source link

SplashAwareFSCacheStorage does not handle HTTP 498 requests #296

Open tmaier opened 2 years ago

tmaier commented 2 years ago

Hi,

I just restarted my crawler and I had HTTP 498 errors, which also in the next tries never worked.

I see the following information has been cached by SplashAwareFSCacheStorage

{"error": 498, "type": "ExpiredArguments", "description": "Arguments stored with ``save_args`` are expired", "info": {"expired": ["lua_source"]}}

It would make sense to specifically handle this case so that the next request will not lead to read this cache (or to not store it at all)