search

pypa / bandersnatch

A PyPI mirror client according to PEP 381 http://www.python.org/dev/peps/pep-0381/
Academic Free License v3.0
454 stars 141 forks source link

Bandersnatch does not exit cleanly when KeyboardInterrupt is triggered #27

Closed dwighthubbard closed 6 years ago

dwighthubbard commented 6 years ago

Bandersnatch generates an exception message and hangs when keyboard interrupt (ctrl-c) is pressed to force it to exit.

Example output, notice the shell prompt does not appear after hitting ctrl-c because bandersnatch did not exit:

$ bandersnatch mirror
^CTraceback (most recent call last):
  File "/var/virtualenv/public_mirror/bin/bandersnatch", line 11, in <module>
    sys.exit(main())
  File "/var/virtualenv/public_mirror/lib/python3.6/site-packages/bandersnatch/main.py", line 105, in main
    args.func(config)
  File "/var/virtualenv/public_mirror/lib/python3.6/site-packages/bandersnatch/main.py", line 60, in mirror
    changed_packages = mirror.synchronize()
  File "/var/virtualenv/public_mirror/lib/python3.6/site-packages/bandersnatch/mirror.py", line 105, in synchronize
    self.sync_packages()
  File "/var/virtualenv/public_mirror/lib/python3.6/site-packages/bandersnatch/mirror.py", line 197, in sync_packages
    package_syncer(packages, self.workers, self.stop_on_error)
  File "/opt/python/lib/python3.6/asyncio/base_events.py", line 454, in run_until_complete
    self.run_forever()
  File "/opt/python/lib/python3.6/asyncio/base_events.py", line 421, in run_forever
    self._run_once()
  File "/opt/python/lib/python3.6/asyncio/base_events.py", line 1395, in _run_once
    event_list = self._selector.select(timeout)
  File "/opt/python/lib/python3.6/selectors.py", line 445, in select
    fd_event_list = self._epoll.poll(timeout, max_ev)
KeyboardInterrupt

Hitting ctrl-c multiple times will eventually cause bandersnatch to exit.

yeraydiazdiaz commented 6 years ago

I took at stab at this but I couldn't find a clean way of doing it. I posted this gist describing the hacky solution running it and pressing Ctrl-C results in a clean exit.

For reference the only solution I found is to:

  1. unregister the atexit registered _python_exit function
  2. call executor.shutdown(wait=False)

The reason is that the thread module registers _python_exit forcing a join of the remaining worker threads

This alone is not enough as shutdown will also join the threads by default so we need to pass wait=False to the call as well.

I could put together a PR implementing it if it's considered necessary but I'm unsure how temporary this asyncio > thread pool executor solution is.

cooperlees commented 6 years ago

This implementation will stay until we replace the XMLRPC API on Warehouse is replaced. #5 and #6 are covering that. With a link to the issue on Warehouse/PyPI.

I don't see that being fast. I'll try nudge people again soon.

yeraydiazdiaz commented 6 years ago

@cooperlees should I create an MR to cover this KeyboardInterrupt issue? when XMLRPC is replaced we could remove it

cooperlees commented 6 years ago

@yeraydiazdiaz Sure sir - Less bugs the better! I want to release 3.0 this week so if we could squeeze it in it would be amazing.