Engine exception: `AttributeError: 'NoneType' object has no attribute 'append'`

[sphuber]

This exception occurs very rarely when a process submits a child process and in an attempt to add itself as a broadcast subscriber, the following exception is thrown:

2019-08-04 20:00:33 [429466 | REPORT]: [879754|PwBaseWorkChain|on_except]: Traceback (most recent call last):
  File "/home/aiida/.virtualenvs/aiida_3dd/lib/python2.7/site-packages/plumpy/process_states.py", line 220, in execute
    result = self.run_fn(*self.args, **self.kwargs)
  File "/home/aiida/code/aiida/env/3dd/aiida-core/aiida/engine/processes/workchains/workchain.py", line 181, in _do_step
    finished, stepper_result = self._stepper.step()
  File "/home/aiida/.virtualenvs/aiida_3dd/lib/python2.7/site-packages/plumpy/workchains.py", line 281, in step
    finished, result = self._child_stepper.step()
  File "/home/aiida/.virtualenvs/aiida_3dd/lib/python2.7/site-packages/plumpy/workchains.py", line 232, in step
    return True, self._fn(self._workchain)
  File "/home/aiida/code/aiida/env/3dd/aiida-quantumespresso/aiida_quantumespresso/workflows/pw/base.py", line 145, in validate_kpoints
    kpoints = create_kpoints_from_distance(**inputs)
  File "/home/aiida/code/aiida/env/3dd/aiida-core/aiida/engine/processes/functions.py", line 197, in decorated_function
    result, _ = run_get_node(*args, **kwargs)
  File "/home/aiida/code/aiida/env/3dd/aiida-core/aiida/engine/processes/functions.py", line 147, in run_get_node
    process = process_class(inputs=inputs, runner=runner)
  File "/home/aiida/.virtualenvs/aiida_3dd/lib/python2.7/site-packages/plumpy/base/state_machine.py", line 188, in __call__
    call_with_super_check(inst.init)
  File "/home/aiida/.virtualenvs/aiida_3dd/lib/python2.7/site-packages/plumpy/base/utils.py", line 29, in call_with_super_check
    fn(*args, **kwargs)
  File "/home/aiida/code/aiida/env/3dd/aiida-core/aiida/engine/processes/process.py", line 134, in init
    super(Process, self).init()
  File "/home/aiida/.virtualenvs/aiida_3dd/lib/python2.7/site-packages/plumpy/base/utils.py", line 16, in new_fn
    fn(self, *args, **kwargs)
  File "/home/aiida/.virtualenvs/aiida_3dd/lib/python2.7/site-packages/plumpy/processes.py", line 290, in init
    self.broadcast_receive, identifier=str(self.pid))
  File "/home/aiida/.virtualenvs/aiida_3dd/lib/python2.7/site-packages/kiwipy/rmq/communicator.py", line 592, in add_broadcast_subscriber
    return self._run_task(coro)
  File "/home/aiida/.virtualenvs/aiida_3dd/lib/python2.7/site-packages/kiwipy/rmq/communicator.py", line 677, in _run_task
    return self.tornado_to_kiwi_future(self._create_task(coro)).result(timeout=self.TASK_TIMEOUT)
  File "/home/aiida/.virtualenvs/aiida_3dd/lib/python2.7/site-packages/kiwipy/rmq/communicator.py", line 656, in tornado_to_kiwi_future
    self.loop().add_future(tornado_future, done)
  File "/home/aiida/.virtualenvs/aiida_3dd/lib/python2.7/site-packages/tornado/ioloop.py", line 597, in add_future
    lambda future: self.add_callback(callback, future))
  File "/home/aiida/.virtualenvs/aiida_3dd/lib/python2.7/site-packages/tornado/concurrent.py", line 270, in add_done_callback
    self._callbacks.append(fn)
AttributeError: 'NoneType' object has no attribute 'append'

This is most likely a bug/problem in kiwipy/tornado but the result is that the parent process excepts and the child process remains in the CREATED state as the required continuation task is not sent to RabbitMQ.

[ltalirz]

@chrisjsewell I vaguely remember we may have fixed something in this direction, do you remember? @sphuber have you encountered this problem recently?

[sphuber]

I have actually not been running AiiDA in quite a while. @mbercx could probably answer this by looking at the recent logs of the 3DCD project. Then again, he wouldn't have been running on a recent version that might have included a fix, so scratch that.

[mbercx]

We're running a custom branch of 1.5.0 for the 3DCD (with the early implementation of the stashing feature, still called archiving at the time), so I wouldn't expect any recent fixes to be included there. I just had a look in my daemon logs for some projects that are running the develop branch of aiida-core though, and haven't found any instances of this issue.