Bad Frame Read

[rhettg]

Just upgraded to librabbitmq 1.5.2 and celery 3.1.12

I'm getting a failure during one of my tasks that involves a large chord.

Stacktrace (most recent call last):

  File "celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "newrelic/hooks/application_celery.py", line 66, in wrapper
    return wrapped(*args, **kwargs)
  File "celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "celery/app/builtins.py", line 75, in unlock_chord
    if deps.ready():
  File "celery/result.py", line 499, in ready
    return all(result.ready() for result in self.results)
  File "celery/result.py", line 499, in 
    return all(result.ready() for result in self.results)
  File "celery/result.py", line 255, in ready
    return self.state in self.backend.READY_STATES
  File "celery/result.py", line 390, in state
    return self._get_task_meta()['status']
  File "celery/result.py", line 328, in _get_task_meta
    meta = self.backend.get_task_meta(self.id)
  File "celery/backends/amqp.py", line 173, in get_task_meta
    accept=self.accept, no_ack=False,
  File "kombu/entity.py", line 570, in get
    message = self.channel.basic_get(queue=self.name, no_ack=no_ack)
  File "librabbitmq/__init__.py", line 65, in basic_get
    frame = self.connection._basic_get(self.channel_id, queue, no_ack)

Task celery.chord_unlock[a360ea41-91c0-4dec-b95f-c1baa3874ce4] raised unexpected: ConnectionError('Bad frame read',)

This seems like it might be related to #30

[rhettg]

I'd appreciate any suggestions on how to further debug this.

[bluebomber]

We're seeing bad frame reads, too but in /librabbitmq/__init__.py, in drain_events.

[armyofevilrobots]

This looks very similar to #30

There is a bug at https://github.com/celery/librabbitmq/blob/master/Modules/_librabbitmq/connection.c#L1343 where an empty (0 byte) body always triggers an exception, even if it was deliberately sent as empty.

I have a patch that fixes this issue in my env in pull request #59.

[allardhoeve]

Any news?

[armyofevilrobots]

That patch #59 has been merged. Try upgrading to 1.6.x and see if you still have the issue.

[allardhoeve]

I believe you missed this merge when releasing 1.6.1:

https://github.com/celery/librabbitmq/commits/v1.6.1 < contains no commit abbd949

Can you please release 1.6.2?

[juliendangers]

:+1: could you please release 1.6.2 ?

[arthurk]

@ask could you release a new version with the fix for this?

[LordNoteworthy]

I have the last version and still getting this Bad Frame Read issue, anyone fixed it ?

[derek-virtustream]

1.6.1 hasn't got this fix IIRC. It's in master and the 1.6 branch though, so checking out either should get around the problem.

[thibaultlavoisey]

Hello,

Any news about 1.6.2?

[seasar]

I get "bad frame read" exception in 1.6.1. maybe I should use other amqp lib instead of librabbitmq.

[derek-virtustream]

It's been a couple years since I issued a patch for this bug. https://github.com/celery/librabbitmq/pull/59#issuecomment-266296125