ireaml
commented
1 year ago Hi @tsthakur, I'm experiencing the same issue.
My environment details are:
- Operating system: MacOS
- Python version: 3.10.9
- aiida-core version: 2.2.2
- RabbitMQ version: 3.11.8
I get a similar report message:
[22m2023-02-21 15:56:14 [3967 | REPORT]: [14092|VaspWorkChain|run_process]: launching VaspCalculation<15027> iteration #1
2023-02-21 16:25:23 [4125 | ERROR]: Traceback (most recent call last):
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/aiida/manage/external/rmq/launcher.py", line 90, in _continue
result = await super()._continue(communicator, pid, nowait, tag)
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/plumpy/process_comms.py", line 604, in _continue
proc = cast('Process', saved_state.unbundle(self._load_context))
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/plumpy/persistence.py", line 58, in unbundle
return Savable.load(self, load_context)
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/plumpy/persistence.py", line 450, in load
return load_cls.recreate_from(saved_state, load_context)
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/plumpy/processes.py", line 244, in recreate_from
call_with_super_check(process.init)
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/plumpy/base/utils.py", line 29, in call_with_super_check
wrapped(*args, **kwargs)
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/aiida/engine/processes/process.py", line 185, in init
super().init()
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/plumpy/base/utils.py", line 16, in wrapper
wrapped(self, *args, **kwargs)
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/plumpy/processes.py", line 303, in init
identifier = self._communicator.add_rpc_subscriber(self.message_receive, identifier=str(self.pid))
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/plumpy/communications.py", line 141, in add_rpc_subscriber
return self._communicator.add_rpc_subscriber(converted, identifier)
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/kiwipy/rmq/threadcomms.py", line 215, in add_rpc_subscriber
return self._loop_scheduler.await_(
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/pytray/aiothreads.py", line 164, in await_
return self.await_submit(awaitable).result(timeout=self.task_timeout)
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/asyncio/tasks.py", line 234, in __step
result = coro.throw(exc)
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/pytray/aiothreads.py", line 178, in coro
res = await awaitable
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/kiwipy/rmq/communicator.py", line 482, in add_rpc_subscriber
identifier = await msg_subscriber.add_rpc_subscriber(subscriber, identifier)
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/kiwipy/rmq/communicator.py", line 123, in add_rpc_subscriber
rpc_queue = await self._channel.declare_queue(exclusive=True, arguments=self._rmq_queue_arguments)
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/aio_pika/robust_channel.py", line 173, in declare_queue
queue = await super().declare_queue(
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/aio_pika/channel.py", line 325, in declare_queue
await queue.declare(timeout=timeout)
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/aio_pika/queue.py", line 92, in declare
self.declaration_result = await asyncio.wait_for(
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
return await fut
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/aiormq/channel.py", line 703, in queue_declare
return await self.rpc(
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/aiormq/base.py", line 168, in wrap
return await self.create_task(func(self, *args, **kwargs))
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/aiormq/base.py", line 25, in __inner
return await self.task
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/asyncio/futures.py", line 285, in __await__
yield self # This tells Task to wait for completion.
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/asyncio/tasks.py", line 304, in __wakeup
future.result()
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/asyncio/futures.py", line 201, in result
raise self._exception.with_traceback(self._exception_tb)
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/asyncio/tasks.py", line 232, in __step
result = coro.send(None)
File "/Users/ireaml/miniconda3/envs/aiida/lib/python3.10/site-packages/aiormq/channel.py", line 121, in rpc
raise ChannelInvalidStateError("writer is None")
aiormq.exceptions.ChannelInvalidStateError: writer is None
Describe the bug
I am not sure if the daemons getting overwhelmed is the reason behind it. But when I launch ~200 calculations together, they get excepted throwing a
aiormq.exceptions.ChannelInvalidStateError: <Channel: "4"> closederror. It is similar to this issue that I opened previously.Following is the full error report
Steps to reproduce
Steps to reproduce the behavior:
Createdstate and start running properly.Expected behavior
Nothing should happen, the workchains should run normally and not get excepted.
Your environment
Additional context
For some reason I am seeing this issue much more frequently now. It used to happen once in a blue moon only if I restarted the daemons, but last time it happened I didn't do anything, the WCs just got excepted after I left the machine alone over the weekend. My environment is still the same, only my aiida database has become bigger.