scrapy / itemloaders

Library to populate items using XPath and CSS with a convenient API
BSD 3-Clause "New" or "Revised" License
44 stars 16 forks source link

Feature Request: Uniform MapCompose-like processor #37

Open Chratho opened 8 years ago

Chratho commented 8 years ago

It took me some time to figure out that MapCompose flattens iterables returned by the specified functions. Even though I then I realized that this behavior is well documented, I still wonder: Why is that? To be honest, it feels only appropriate for highly-specific use-cases. It works fine for unary values, but if people start using it on multi-valued fields (e.g., tuples) the results might be counter-intuitive.

The following code is a slightly adapted version of MapCompose, mostly just using append instead of += for value-concatenation:

class ListCompose(object):

def __init__(self, *functions, **default_loader_context):
    self.functions = functions
    self.default_loader_context = default_loader_context

def __call__(self, values, loader_context=None):
    if loader_context:
        context = MergeDict(loader_context, self.default_loader_context)
    else:
        context = self.default_loader_context
    wrapped_funcs = [wrap_loader_context(f, context) for f in self.functions]
    for func in wrapped_funcs:
        next_values = []
        for v in values:
            next_values.append(func(v))
        values = next_values
    return values

Since nearly my all fields are initially created by add_xpath (an approach where Compose turns out to be pointless), I ended up using ListCompose a lot (just like MapCompose it works for unary fields as well