search

scipy-conference / procbuild

SciPy proceedings builder
Other
2 stars 10 forks source link

Queue fix #5

Closed katyhuff closed 8 years ago

katyhuff commented 8 years ago

When tracking down build issue, I ran accross a known issue in which multiprocessing.Queue.qsize() is NotImplemented on Mac.

I was getting one of these:

ERROR:procbuild:Exception on /build/0 [GET]
Traceback (most recent call last):
  File "/Users/huff/anaconda3/envs/py27/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/huff/anaconda3/envs/py27/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/huff/anaconda3/envs/py27/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/huff/anaconda3/envs/py27/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/huff/anaconda3/envs/py27/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/huff/procbuild/procbuild/server.py", line 112, in build
    return real_build(*args, **kwarg)
  File "/Users/huff/procbuild/procbuild/server.py", line 96, in real_build
    if paper_queue.qsize() >= 50:
  File "/Users/huff/anaconda3/envs/py27/lib/python2.7/multiprocessing/queues.py", line 143, in qsize
    return self._maxsize - self._sem._semlock._get_value()
NotImplementedError

So I went with a solution similar to this one: https://sourceforge.net/p/blockit/support-requests/1/

Also, I slipped in an unrelated change to the readme to support PR in scipy_proceedings: https://github.com/scipy-conference/scipy_proceedings/pull/190

stefanv commented 8 years ago

This fix never decreases the queue size, so after 50 papers have been built, the builder gets stuck.