Request fetching from `RequestQueue` is sometimes very slow

[vdusek]

• Fetching requests from RequestQueue is sometimes very slow and can get stuck for a while.
• I turned on logging and reproduced the issue with the following code:

import asyncio
import logging

from crawlee.beautifulsoup_crawler import BeautifulSoupCrawler, BeautifulSoupCrawlingContext

logging.basicConfig(level=logging.INFO)

async def main() -> None:
    crawler = BeautifulSoupCrawler()

    @crawler.router.default_handler
    async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
        await context.enqueue_links(strategy='same-hostname')
        data = {
            'request_url': context.request.url,
            'soup_url': context.soup.url,
            'soup_title': context.soup.title.string if context.soup.title else None,
        }
        await context.push_data(data)

    await crawler.run(['https://crawlee.dev'])

if __name__ == '__main__':
    asyncio.run(main())

• In the logs, there are many lines like this:

  INFO:crawlee.storages.request_queue:Waiting for 9.988466 for queue finalization, to ensure data consistency.

• This logging is from the following code block: crawlee/storages/request_queue.py#L541:L546

Questions

• Is this behavior correct?
• Is the waiting period necessary?
• Is it necessary for memory storage as well?

[vdusek]

In https://github.com/apify/crawlee-python/pull/235 we set ensure_consistency to False as a hotfix in the request queue.

The root cause of this behavior is in https://github.com/apify/crawlee-python/pull/186, before that, the RQ works fine, without waiting. Let's deep dive into that after the public launch and documentation is ready.

Code to reproduce it:

import asyncio
import logging

from crawlee.beautifulsoup_crawler import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee.storages import Dataset
from crawlee.log_config import CrawleeLogFormatter

# Configuration the logging
handler = logging.StreamHandler()
handler.setFormatter(CrawleeLogFormatter(include_logger_name=True))
root_logger = logging.getLogger()
root_logger.setLevel(logging.INFO)
root_logger.addHandler(handler)

async def main() -> None:
    crawler = BeautifulSoupCrawler()
    dataset = await Dataset.open()

    @crawler.router.default_handler
    async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
        await context.enqueue_links()
        data = {
            'url': context.request.url,
            'title': context.soup.title.string if context.soup.title else '',
        }
        await dataset.push_data(data)

    await crawler.run(['https://crawlee.dev/'])

if __name__ == '__main__':
    asyncio.run(main())