uvacw / inca

24 stars 6 forks source link

Make processors more similar to scrapers #424

Open damian0604 opened 6 years ago

damian0604 commented 6 years ago

The .runwrap() method of scrapers, well, simply runs scrapers. So, one can do:

myinca.rssscrapers.nu()

and that's it.

With processors, instead, .runwrap() returns a generator. That's great for save=False, but if we simply want to run the processor and store the result in the database, we have to use a cumbersome construction like:

r = [e for e in myinca.processing.regex_tagger('nu','text',regex = r'DENK', new_key='denk')]

or

p = myinca.processing.regex_tagger('nu','text',regex = r'DENK', new_key='denk')
while True:
    next(p)

We should investigate whether it makes sense to have processors behave more like scrapers