search

newslynx / opportunities

A garden of NewsLynx futures
1 stars 0 forks source link

Deal with pickling issue with blueprint loader / rq worker / bulk content item and event creation #27

Closed abelsonlive closed 9 years ago

abelsonlive commented 9 years ago

There is currently a pretty nasty bug in newslynx-core where rq fails to properly execute bulk event creation because of some Import Error that is raised when pickling the worker class. This error is being thrown within the code that dynamically builds up the API by crawling the views/ folder for all Blueprints. I have spent a couple late nights trying to get around this, but can't figure it out. For now I patched it here:

https://github.com/newslynx/newslynx-core/blob/master/newslynx/views/util.py#L652

However, this is undesirable since if there are syntax errors in a blueprint, they will be silently ignored which effectively removes that blueprint from the API. If you're not aware of this error, then you will spent a long time banging your head against the wall trying to figure out why you can't access a particular endpoint.

As far as I can tell, bulk loading works fine except for this.

abelsonlive commented 9 years ago

The long-run solution should be switching to celery

abelsonlive commented 9 years ago

I dealt with this issue by completely rewriting ingestion to be classless, and thereby enable pickling.