search

mvantellingen / localshop

local pypi server (custom packages and auto-mirroring of pypi)
MIT License
397 stars 112 forks source link

Can't start (localshop celeryd -B -E fails) #21

Closed cakebread closed 11 years ago

cakebread commented 12 years ago

I've tried installing localshop in a virtualenv and directly, both result in the same traceback. I'm able to import the localshop module from the command-line with "python -c 'import localshop'"

$ localshop celeryd -B -E
/usr/local/lib/python2.6/dist-packages/django/conf/__init__.py:110: DeprecationWarning: The SECRET_KEY setting must not be empty.
  warnings.warn("The SECRET_KEY setting must not be empty.", DeprecationWarning)

 -------------- celery@pages02 v3.0.0 (Chiastic Slide)
---- **** ----- 
--- * ***  * -- [Configuration]
-- * - **** --- . broker:      django://localhost//
- ** ---------- . app:         default:0x19d55d0 (djcelery.loaders.DjangoLoader)
- ** ---------- . concurrency: 1 (processes)
- ** ---------- . events:      ON
- ** ---------- 
- *** --- * --- [Queues]
-- ******* ---- . celery:      exchange:celery(direct) binding:celery
--- ***** ----- 

[2012-07-09 18:28:44,483: ERROR/MainProcess] Unrecoverable error: PicklingError("Can't pickle 'lock' object: <thread.lock object at 0x260f210>",)
Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/celery/worker/__init__.py", line 350, in start                                                                                      
    component.start()                                                                                                                                                              
  File "/usr/local/lib/python2.6/dist-packages/billiard/process.py", line 138, in start                                                                                            
    self._popen = Popen(self)                                                                                                                                                      
  File "/usr/local/lib/python2.6/dist-packages/billiard/forking.py", line 223, in __init__                                                                                         
    dump(process_obj, to_child, HIGHEST_PROTOCOL)                                                                                                                                  
  File "/usr/local/lib/python2.6/dist-packages/billiard/forking.py", line 143, in dump                                                                                             
    ForkingPickler(file, protocol).dump(obj)                                                                                                                                       
  File "/usr/lib/python2.6/pickle.py", line 224, in dump                                                                                                                           
    self.save(obj)                                                                                                                                                                 
  File "/usr/lib/python2.6/pickle.py", line 331, in save                                                                                                                           
    self.save_reduce(obj=obj, *rv)                                                                                                                                                 
  File "/usr/lib/python2.6/pickle.py", line 419, in save_reduce                                                                                                                    
    save(state)                                                                                                                                                                    
  File "/usr/lib/python2.6/pickle.py", line 286, in save                                                                                                                           
    f(self, obj) # Call unbound method with explicit self                                                                                                                          
  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict                                                                                                                      
    self._batch_setitems(obj.iteritems())                                                                                                                                          
  File "/usr/lib/python2.6/pickle.py", line 681, in _batch_setitems                                                                                                                
    save(v)                                                                                                                                                                        
  File "/usr/lib/python2.6/pickle.py", line 331, in save                                                                                                                           
    self.save_reduce(obj=obj, *rv)                                                                                                                                                 
  File "/usr/lib/python2.6/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.6/pickle.py", line 681, in _batch_setitems
    save(v)
  File "/usr/lib/python2.6/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.6/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.6/pickle.py", line 681, in _batch_setitems
    save(v)
  File "/usr/lib/python2.6/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.6/pickle.py", line 419, in save_reduce
    save(state)
  File "/usr/lib/python2.6/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.6/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.6/pickle.py", line 681, in _batch_setitems
    save(v)
  File "/usr/lib/python2.6/pickle.py", line 313, in save
    (t.__name__, obj))
PicklingError: Can't pickle 'lock' object: <thread.lock object at 0x260f210>
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.6/dist-packages/billiard/forking.py", line 470, in main
    prepare(preparation_data)
  File "/usr/local/lib/python2.6/dist-packages/billiard/forking.py", line 610, in prepare
    file, path_name, etc = imp.find_module(main_name, dirs)
ImportError: No module named localshop
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.6/dist-packages/billiard/forking.py", line 470, in main
    prepare(preparation_data)
  File "/usr/local/lib/python2.6/dist-packages/billiard/forking.py", line 610, in prepare
    file, path_name, etc = imp.find_module(main_name, dirs)
ImportError: No module named localshop
007 commented 12 years ago

Same error here, on Ubuntu 10.04 / Python 2.6.

Haven't tested outside virtualenv, but fails within virtualenv.

stuaxo commented 11 years ago

Just found the same bug, this definitely fixes it.

stuaxo commented 11 years ago

OK, spoke to soon - django-celery==2.5.5 fixed a problem with this: https://github.com/celery/celery/issues/683

So 2.5.5 for celery AND django-celery

jezdez commented 11 years ago

As a work around you can use localshop celeryd -B -E -P solo which is a single process worker implementation.

mvantellingen commented 11 years ago

Thanks, I actually have the same issue. Not yet sure how to fix this yet (or what is causing this). Perhaps an incompatibility between logan and celery

mvantellingen commented 11 years ago

This should now be fixed in the develop branch (thanks @jezdez)

victorhooi commented 11 years ago

Hi,

I've just checked out the latest develop branch of localshop, and I seem to still be getting this issue:

localshop celeryd -B -E
Notice: Unable to load configuration file /home/vichoo/.localshop.py (No such file or directory), using default settings

 -------------- celery@hostname.com v3.0.12 (Chiastic Slide)
---- **** ----- 
--- * ***  * -- [Configuration]
-- * - **** --- . broker:      django://localhost//
- ** ---------- . app:         default:0x18d1c3d0 (djcelery.loaders.DjangoLoader)
- ** ---------- . concurrency: 1 (processes)
- ** ---------- . events:      ON
- ** ---------- 
- *** --- * --- [Queues]
-- ******* ---- . celery:      exchange:celery(direct) binding:celery
--- ***** ----- 

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 502, in main
    prepare(preparation_data)
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 642, in prepare
    file, path_name, etc = imp.find_module(main_name, dirs)
ImportError: No module named localshop
[2013-01-02 05:24:14,366: WARNING/MainProcess] celery@dev_desktop_vm.aus.optiver.com ready.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 502, in main
    prepare(preparation_data)
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 642, in prepare
    file, path_name, etc = imp.find_module(main_name, dirs)
ImportError: No module named localshop
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 502, in main
    prepare(preparation_data)
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 642, in prepare
    file, path_name, etc = imp.find_module(main_name, dirs)
ImportError: No module named localshop
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 502, in main
    prepare(preparation_data)
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 642, in prepare
    file, path_name, etc = imp.find_module(main_name, dirs)
ImportError: No module named localshop
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 502, in main
    prepare(preparation_data)
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 642, in prepare
    file, path_name, etc = imp.find_module(main_name, dirs)
ImportError: No module named localshop
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 502, in main
    prepare(preparation_data)
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 642, in prepare
    file, path_name, etc = imp.find_module(main_name, dirs)
ImportError: No module named localshop
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 502, in main
    prepare(preparation_data)
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 642, in prepare
    file, path_name, etc = imp.find_module(main_name, dirs)
ImportError: No module named localshop
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 502, in main
    prepare(preparation_data)
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 642, in prepare
    file, path_name, etc = imp.find_module(main_name, dirs)
ImportError: No module named localshop
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 502, in main
    prepare(preparation_data)
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 642, in prepare
    file, path_name, etc = imp.find_module(main_name, dirs)
ImportError: No module named localshop
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 502, in main
    prepare(preparation_data)
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 642, in prepare
    file, path_name, etc = imp.find_module(main_name, dirs)
ImportError: No module named localshop
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 502, in main
    prepare(preparation_data)
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 642, in prepare
    file, path_name, etc = imp.find_module(main_name, dirs)
ImportError: No module named localshop
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 502, in main
    prepare(preparation_data)
  File "/home/vichoo/.virtualenvs/localshop/lib/python2.7/site-packages/billiard-2.7.3.19-py2.7-linux-x86_64.egg/billiard/forking.py", line 642, in prepare
    file, path_name, etc = imp.find_module(main_name, dirs)
ImportError: No module named localshop

I can confirm that using localshop celeryd -B -E -P solo, as jezdez mentions seems to work - as in, it doesn't generate any errors on the console. I've yet to test actually downloading packages =).

Any thoughts on why this is still happening?

Cheers, Victor

mvantellingen commented 11 years ago

Oke, well I can confirm that it still doesn't work. I'm going to look for a celery alternative...

jezdez commented 11 years ago

Hm, it works for me. Can you try python manage.py celeryd -B -E instead? Smells like Celery just stumbles over not finding the right binary to spawn the workers with.

mvantellingen commented 11 years ago

Well i've just created a fresh virtualenv, ran python setup.py develop and then ./manage.py -B -E -c 1 but no luck.

django.core.exceptions.ImproperlyConfigured: django-configurations settings importer wasn't correctly installed. Please use one of the starter functions to install it as mentioned in the docs: http://django-configurations.readthedocs.org/

Will look at it some more tonight

On Wed, Jan 2, 2013 at 5:04 PM, Jannis Leidel notifications@github.comwrote:

Hm, it works for me. Can you try python manage.py celeryd -B -E instead? Smells like Celery just stumbles over not finding the right binary to spawn the workers with.

— Reply to this email directly or view it on GitHubhttps://github.com/mvantellingen/localshop/issues/21#issuecomment-11812938.

jezdez commented 11 years ago

Make sure you use the manage.py that is included, as it requires to use the tools in configurations to work: https://github.com/mvantellingen/localshop/blob/develop/localshop/runner.py

mvantellingen commented 11 years ago

Yup, that's the one I've used. I just removed the usage of django-configurations as a test and then it works. So it looks like celery does something weird which causes problem with logan and django-configurations.

@jezdez Could you retry in a fresh virtualenv?

jezdez commented 11 years ago

@mvantellingen did you push it to develop?

mvantellingen commented 11 years ago

@jezdez yup. But I think I found the issue. The procfile contains '--no-execv'. With this option everything works. Without it you should also get the error

jezdez commented 11 years ago

@mvantellingen Yeah, it seems like this is an issue with celery, in case you want to set this in general you could also use CELERYD_FORCE_EXECV = False

mvantellingen commented 11 years ago

Yup already decided to just add it and be done with it :-) Thanks