refactor of client's scraping modules management thanks to a brain.think function
prepare_batch sizing can be defined with a function which takes an UnprocessedItem and return an integer
scrapping modules changes :
scraping modules query prototype has been updated to take a positional argument parameters:dict to allow behavior specification from within the scraping module manager
generate_url has been moved inside the module responsibilities
internal client changes :
brain.think
functionprepare_batch
sizing can be defined with a function which takes anUnprocessedItem
and return aninteger
scrapping modules changes :
parameters:dict
to allow behavior specification from within the scraping module managergenerate_url
has been moved inside the module responsibilities