Don’t include (if possible) batches of requests containing requests for specific host if there are already more then specified count of maximum requests per host. This is a suggestion for broad crawling queue get algorithm.
Hi, these requests are skipped for batch of results prepared for specific iteration. They will stay in HBase table for subsequent runs. Only requests added to trash_can are removed completely.
Hi, BC_MAX_REQUESTS_PER_HOST description states:
However in practice, the urls exceeding specified count of maximum requests per host are dropped for a given row key. I would expect such urls to be included in later batches, not dropped. This is the part of the code in question: https://github.com/scrapinghub/frontera/blob/master/frontera/contrib/backends/hbase/__init__.py#L249
Is this expected behavior?
In case it is, this should be explicitly stated in the documentation to avoid possible misuse of the option.
Best regards