search

scrapinghub / frontera

A scalable frontier for web crawlers
BSD 3-Clause "New" or "Revised" License
1.29k stars 216 forks source link

Addition: Docs need to say that _refresh_states() also takes a non iterable #354

Open IAlwaysBeCoding opened 5 years ago

IAlwaysBeCoding commented 5 years ago

The docs need to be changed to let the user know that _refresh_states() also takes a non iterable, aka a single object not contained in a list.

https://github.com/scrapinghub/frontera/blob/8ac23162e97f6e685534c029e1da19d867c037d7/frontera/strategy/__init__.py#L139-L145

That's because the function refresh_and_keep() converts the non iterable object into a non-iterable object contained within a list.

https://github.com/scrapinghub/frontera/blob/8ac23162e97f6e685534c029e1da19d867c037d7/frontera/core/manager.py#L818-L822

sibiryakov commented 5 years ago

Good catch! Can you make PR @IAlwaysBeCoding ?