search

pgq / skytools-legacy

Obsolete, see https://github.com/pgq/ for maintained code.
https://github.com/pgq/
Other
248 stars 86 forks source link

KeyError with CoopConsumer and pgq_lazy_fetch=0 #28

Open epeld opened 11 years ago

epeld commented 11 years ago

I get a Traceback (most recent call last): File "/home/avtech/skytools/python/skytools/scripting.py", line 568, in run_func_safely return func() File "/home/avtech/skytools/python/pgq/baseconsumer.py", line 260, in work self._finish_batch(curs, batch_id, ev_list) File "/home/avtech/skytools/python/pgq/coopconsumer.py", line 76, in _finish_batch self._flush_retry(curs, batch_id, list) File "/home/avtech/skytools/python/pgq/consumer.py", line 76, in _flush_retry if ev._status == EV_RETRY: File "/home/avtech/skytools/python/pgq/event.py", line 49, in getattr return self._event_row[_fldmap[key]] KeyError: '_status' when running a CoopConsumer with pgq_lazy_fetch=0.

I believe that the cause of the problem is that pgq.baseconsumer.py:_load_batch_events_old() instantiates events directly using the Event-constructor, but CoopConsumer expects "RetriableEvent"s instead. That is, the Event-instances do not have a _status-field whereas RetriableEvents do.

When pgq_lazy_fetch > 1 the events are instantiated by the _batch_walker_class-field in _load_batch_events() so there is no crash (because CoopConsumer supplies its own BatchWalker which creates RetriableEvents).

Maybe events should always be created by some function _make_event that subclasses can override?

markokr commented 11 years ago

Thanks for report and good analysis. Fixed in GIT with _make_event() in consumer classes.

Alternative would be to have _event_class, but that is not good unless shared somehow with walker. So I kept things simple ATM.

Please test.

epeld commented 11 years ago

Hi, thanks for the quick fix.

Unfortunately there are still problems: Traceback (most recent call last): File "/usr/local/lib64/python2.6/site-packages/skytools/scripting.py", line 568, in run_func_safely return func() File "/usr/local/lib64/python2.6/site-packages/pgq/baseconsumer.py", line 257, in work self._launch_process_batch(db, batch_id, ev_list) File "/usr/local/lib64/python2.6/site-packages/pgq/baseconsumer.py", line 286, in _launch_process_batch self.process_batch(db, batch_id, list) File "/usr/local/lib64/python2.6/site-packages/pgq/baseconsumer.py", line 233, in process_batch self.process_event(db, ev) File "./av2te_consumer.py", line 166, in process_event ev.tag_done() File "/usr/local/lib64/python2.6/site-packages/pgq/consumer.py", line 23, in tag_done self._walker.tag_event_done(self) AttributeError: 'AV2TEConsumer' object has no attribute 'tag_event_done'

I believe the problem is due to this line: https://github.com/markokr/skytools/blob/master/python/pgq/consumer.py#L64 which attempts to create a RetriableWalkerEvent by passing self as the walker-parameter. However self here is a Consumer-instance so does not have a tag_event_done method. I think you meant to pass an instance of RetriableBatchWalker instead of self?

markokr commented 11 years ago

we don't have walker there. The right fix is to use RetryableEvent.

I did some reorg too, to make stuff more obvious next time.

Please test.