pickling error: attribute lookup CastTaggedQuerySet on tagulous.models.tagged failed

[valentijnscholten]

Just a quick initial description here, I am still seeing the pickling errors with 1.3.0.

Traceback (most recent call last):
  File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 42, in _reraise_errors
    yield
  File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 213, in dumps
    payload = encoder(data)
  File "/home/valentijn/venv/lib/python3.8/site-packages/kombu/serialization.py", line 337, in pickle_dumps
    return dumper(obj, protocol=pickle_protocol)
_pickle.PicklingError: Can't pickle <class 'tagulous.models.tagged.CastTaggedQuerySet'>: attribute lookup CastTaggedQuerySet on tagulous.models.tagged failed

What can I provide to help debugging?

[valentijnscholten]

BTW the errors looks different from #109

[valentijnscholten]

Still happening. Basically if this doesn't work it means we cannot use celery in our application. Or we cannot use django-tagulous in our application.

INFO [celery.worker.strategy:155] Task dojo.notifications.helper.send_slack_notification[99ad2c16-2586-4868-8d5f-109176b6ec61] received
kombu.exceptions.EncodeError: Can't pickle <class 'tagulous.models.tagged.CastTaggedQuerySet'>: attribute lookup CastTaggedQuerySet on tagulous.models.tagged failed
"return dumper(obj, protocol=pickle_protocol)"
"File ""/usr/local/lib/python3.8/site-packages/kombu/serialization.py"", line 334, in pickle_dumps"
payload = encoder(data)
"File ""/usr/local/lib/python3.8/site-packages/kombu/serialization.py"", line 210, in dumps"
yield
"File ""/usr/local/lib/python3.8/site-packages/kombu/serialization.py"", line 39, in _reraise_errors"
raise value.with_traceback(tb)
"File ""/usr/local/lib/python3.8/site-packages/kombu/exceptions.py"", line 21, in reraise"
"reraise(wrapper, wrapper(exc), sys.exc_info()[2])"
"File ""/usr/local/lib/python3.8/site-packages/kombu/serialization.py"", line 43, in _reraise_errors"
"self.gen.throw(type, value, traceback)"
"File ""/usr/local/lib/python3.8/contextlib.py"", line 131, in __exit__"
payload = encoder(data)
INFO [celery.worker.strategy:155] Task dojo.notifications.helper.send_slack_notification[99ad2c16-2586-4868-8d5f-109176b6ec61] received
kombu.exceptions.EncodeError: Can't pickle <class 'tagulous.models.tagged.CastTaggedQuerySet'>: attribute lookup CastTaggedQuerySet on tagulous.models.tagged failed
"return dumper(obj, protocol=pickle_protocol)"
"File ""/usr/local/lib/python3.8/site-packages/kombu/serialization.py"", line 334, in pickle_dumps"
payload = encoder(data)
"File ""/usr/local/lib/python3.8/site-packages/kombu/serialization.py"", line 210, in dumps"
yield
"File ""/usr/local/lib/python3.8/site-packages/kombu/serialization.py"", line 39, in _reraise_errors"
raise value.with_traceback(tb)
"File ""/usr/local/lib/python3.8/site-packages/kombu/exceptions.py"", line 21, in reraise"
"reraise(wrapper, wrapper(exc), sys.exc_info()[2])"
"File ""/usr/local/lib/python3.8/site-packages/kombu/serialization.py"", line 43, in _reraise_errors"
"self.gen.throw(type, value, traceback)"
"File ""/usr/local/lib/python3.8/contextlib.py"", line 131, in __exit__"
payload = encoder(data)INFO [celery.worker.strategy:155] Task dojo.notifications.helper.send_slack_notification[99ad2c16-2586-4868-8d5f-109176b6ec61] received
kombu.exceptions.EncodeError: Can't pickle <class 'tagulous.models.tagged.CastTaggedQuerySet'>: attribute lookup CastTaggedQuerySet on tagulous.models.tagged failed
"return dumper(obj, protocol=pickle_protocol)"
"File ""/usr/local/lib/python3.8/site-packages/kombu/serialization.py"", line 334, in pickle_dumps"
payload = encoder(data)
"File ""/usr/local/lib/python3.8/site-packages/kombu/serialization.py"", line 210, in dumps"
yield
"File ""/usr/local/lib/python3.8/site-packages/kombu/serialization.py"", line 39, in _reraise_errors"
raise value.with_traceback(tb)
"File ""/usr/local/lib/python3.8/site-packages/kombu/exceptions.py"", line 21, in reraise"
"reraise(wrapper, wrapper(exc), sys.exc_info()[2])"
"File ""/usr/local/lib/python3.8/site-packages/kombu/serialization.py"", line 43, in _reraise_errors"
"self.gen.throw(type, value, traceback)"
"File ""/usr/local/lib/python3.8/contextlib.py"", line 131, in __exit__"
payload = encoder(data)INFO [celery.worker.strategy:155] Task dojo.notifications.helper.send_slack_notification[99ad2c16-2586-4868-8d5f-109176b6ec61] received
kombu.exceptions.EncodeError: Can't pickle <class 'tagulous.models.tagged.CastTaggedQuerySet'>: attribute lookup CastTaggedQuerySet on tagulous.models.tagged failed
"return dumper(obj, protocol=pickle_protocol)"
"File ""/usr/local/lib/python3.8/site-packages/kombu/serialization.py"", line 334, in pickle_dumps"
payload = encoder(data)
"File ""/usr/local/lib/python3.8/site-packages/kombu/serialization.py"", line 210, in dumps"
yield
"File ""/usr/local/lib/python3.8/site-packages/kombu/serialization.py"", line 39, in _reraise_errors"
raise value.with_traceback(tb)
"File ""/usr/local/lib/python3.8/site-packages/kombu/exceptions.py"", line 21, in reraise"
"reraise(wrapper, wrapper(exc), sys.exc_info()[2])"
"File ""/usr/local/lib/python3.8/site-packages/kombu/serialization.py"", line 43, in _reraise_errors"
"self.gen.throw(type, value, traceback)"
"File ""/usr/local/lib/python3.8/contextlib.py"", line 131, in __exit__"
payload = encoder(data)

[radiac]

Apologies for the time it's taking to sort this.

In #143 we're talking about removing the tag manager from pickling altogether - I'm hopeful this would be another way to fix this issue, but need to test.

I'm also a little concerned that it might be that kombu is pickling in a way I'm not testing, or it's pickling an unexpected object.

I suggest I deal with #143 first, then perhaps you could try that change out on your code and see if it solves the problem. If it doesn't, I may need some help coming up with a relatively simple test case to recreate the error - but we can cross that bridge in a bit if we need to

[radiac]

Looking at the error it seems you're pickling a queryset, so the change for #143 isn't related. The cast classes for managers and querysets weren't ever attached to a module, so couldn't be imported. I've updated the casting code to house the cast class back in the originating module, so this should now be resolved.

As an aside, it may be worth considering passing object pks into celery rather than pickling the objects themselves. Looking the objects up again fresh from the db means you never miss db changes and unpickle stale or inconsistent data.

That said, hopefully this will now work, and do let me know if you run into any more issues.