search

medialab / quenouille

A library of multithreaded iterator workflows for python.
MIT License
7 stars 0 forks source link

Issue when using throttle with group_parallelism #62

Closed Yomguithereal closed 1 year ago

Yomguithereal commented 1 year ago 
Traceback (most recent call last):
  File "/Users/Yomgui/.pyenv/versions/3.7.10/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/Yomgui/.pyenv/versions/3.7.10/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/Yomgui/code/shelob/shelob/cli/__main__.py", line 11, in <module>
    main()
  File "/Users/Yomgui/code/shelob/shelob/cli/__main__.py", line 7, in main
    run("shelob", "%s (%s)" % (__version__, __published__), SHELOB_COMMANDS)
  File "/Users/Yomgui/.pyenv/versions/shelob/lib/python3.7/site-packages/minet/cli/utils.py", line 48, in wrapper
    fn(*args, **kwargs)
  File "/Users/Yomgui/.pyenv/versions/shelob/lib/python3.7/site-packages/minet/cli/run.py", line 117, in run
    fn(cli_args)
  File "/Users/Yomgui/.pyenv/versions/shelob/lib/python3.7/site-packages/minet/cli/utils.py", line 326, in wrapper
    return action(cli_args, *args, **additional_kwargs, **kwargs)
  File "/Users/Yomgui/code/shelob/shelob/cli/lemonde/crawl.py", line 52, in action
    for result in crawler:
  File "/Users/Yomgui/.pyenv/versions/shelob/lib/python3.7/site-packages/minet/crawl/crawler.py", line 356, in safe_wrapper
    for result in imap_unordered:
  File "/Users/Yomgui/.pyenv/versions/shelob/lib/python3.7/site-packages/quenouille/imap.py", line 959, in output
    buffer.unregister_job(job)
  File "/Users/Yomgui/.pyenv/versions/shelob/lib/python3.7/site-packages/quenouille/imap.py", line 498, in unregister_job
    self.throttled_groups.add(job)
  File "/Users/Yomgui/.pyenv/versions/shelob/lib/python3.7/site-packages/quenouille/imap.py", line 236, in add
    assert group not in self
AssertionError