search

scrapy / itemloaders

Library to populate items using XPath and CSS with a convenient API
BSD 3-Clause "New" or "Revised" License
44 stars 16 forks source link

Optimizing wrap_loader_context() #49

Open wRAR opened 2 years ago

wRAR commented 2 years ago

When working on a loader-heavy project I found that a lot of various inspect calls (and just a lot of various function calls) is done for every field of every loader. wrap_loader_context() calls get_func_args() for each processor which in turn does the most of the aforementioned things.

My tests show that a simple @lru_cache(1024) for get_func_args() is enough and that the cache in this case will contain one or several items for each processor used by the spider (so 1024 should be enough, though if it isn't, the cache will fail to give the performance improvements so this is debatable).

Another option would be changing get_value() and get_output_value() so that wrap_loader_context() isn't called this often, but get_value() takes the processors as input values.

wRAR commented 2 years ago

Related: https://github.com/scrapy/scrapy/pull/2889

soundofspace commented 1 year ago

Any reason to not merge a lru cache implementation? I came here after I noticed huge performance decrease for some spiders after updating to v1.1.0 from v1.0.6.

Some metrics for the most affected spider I could find: Version items/second
v1.0.6 350
v1.1.0 120
v1.1.0 with lru 550
image