inspirehep / hepcrawl

Scrapy project for feeds into INSPIRE-HEP
http://inspirehep.net
Other
17 stars 30 forks source link

utils: strictly check kwargs in spiders (#218) #219

Closed szymonlopaciuk closed 6 years ago

szymonlopaciuk commented 6 years ago

This introduced a decorator which will disallow any keyword arguments passed to the decorated method that do not begin with an underscore sign in the decorated method. This is mainly to make errors while passing arguments to spiders immediately visible. As we cannot remove kwargs from the spiders' constructors altogether (used internally by scrapyd), with this we can ensure that we are not passing unwanted data by mistake.

Additionaly this will add all of the 'public' kwargs to an self._init_kwargs field in the object for easier passing and all of the arguments (including non-overloaded ones) to self._all_kwargs. (To make passing them forward easier.)