python 3.8 issue

[gaborbernat]

For a service that works perfectly under python3.7 fails under python 3.8:

Traceback (most recent call last):
  File "/.tox/py38/bin/huey_consumer", line 8, in <module>
    sys.exit(consumer_main())
  File "/.tox/py38/lib/python3.8/site-packages/huey/bin/huey_consumer.py", line 51, in consumer_main
    consumer.run()
  File "/.tox/py38/lib/python3.8/site-packages/huey/consumer.py", line 404, in run
    self.start()
  File "/.tox/py38/lib/python3.8/site-packages/huey/consumer.py", line 373, in start
    self.scheduler.start()
  File "/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'Consumer._create_process.<locals>._run'

[coleifer]

Can you provide a little more information so I can try and replicate this? The tests are run against 3.8: https://travis-ci.org/coleifer/huey/jobs/616785290

[coleifer]

The example (using processes) seems to be running just fine under 3.8:

If you can provide more information so I can replicate or debug this, I will gladly reopen.

[gaborbernat]

I've ran into this with pyenv on Mac. 🤔 It still does the above thing for me.

[coleifer]

I think it may be related: https://bugs.python.org/issue33884 -- which was closed. I'd suggest making a small modification in your code to try and explicitly set the multiprocessing start mode to 'fork':

https://docs.python.org/3/library/multiprocessing.html#multiprocessing.set_start_method

You can put it alongside where you declare your huey instance.

See also here: https://github.com/sphinx-doc/sphinx/issues/6803#issuecomment-554704033

[coleifer]

Added workaround to consumer entry-point: bf002c85ae444954c3237fa4b056fbf9111448a1

[seemab-yamin]

@coleifer I am facing the same on MAC Pyhton 3.11.5

[coleifer]

For some reason your environment is using the spawn method. There is a workaround in the huey_consumer.py entrypoint here, but it looks like you are not using it: https://github.com/coleifer/huey/blob/master/huey/bin/huey_consumer.py#L60-L65

Just add the code from above to whatever you are using for your entrypoint to your application and you should be set.