Tasks with autoretry_for stuck in locked state

[andbortnik]

I've discovered a bug in get_key method which prevents lock release.

@app.task(name='mytask', base=celery_once.QueueOnce)
def mytask(foo, bar):
    ...

mytask.delay(foo=1, bar=2)

The getcallargs(self.run, *args, **kwargs) call returns {'foo': 1, 'bar': 2}. The get_key call returns qo_mytask_foo-1_bar-2. Everything works properly.

If I'd add autoretry_for argument:

@app.task(name='mytask_retry', base=celery_once.QueueOnce, autoretry_for=(Exception, ))
def mytask_retry(foo, bar):
    ...

mytask_retry.delay(foo=1, bar=2)

The getcallargs(self.run, *args, **kwargs) call returns {'args': (), 'kwargs': {'foo': 1, 'bar': 2}}. The get_key call returns qo_mytask_retry_args-()_kwargs={'bar': 2, 'foo': 1}. The {'bar': 2, 'foo': 1} part is unsorted and sometimes the order changes to {'foo': 1, 'bar': 2}.

The same issue takes place when a nested dict is passed as task's argument. For example, mytask(foo={'spam': 10, 'eggs': 20}, bar=2). The get_key call returns qo_mytask_foo-{'eggs': 20, 'spam': 10}_bar-2 and sometimes qo_mytask_foo-{'spam': 10, 'eggs': 20}_bar-2

This lead to lock release issues because get_key result is different on lock stage and release stage.