BatchTimeoutError while running the SSH example

[ederenn]

Name: blue yagna version: yagna 0.10.0-rc15 (160bc5a1 2022-03-14 build #206) OS+lang+version (if applicable): mac, python 3.9, yapapi 0.9.0-alpha.1

[<SshService starting on jiuzhang.t [ 0x02a71376b982cb3752e99252625aa4cc33b7ed3d ] @ 192.168.0.8>, <SshService terminated on 2rec-ubuntu.t [ 0x1b6505b1edba89ead6767305fff17a6eff09416c ] @ 192.168.0.2>]
[2022-03-15T13:26:29.372+0100 WARNING yapapi.services.service_runner] Unhandled exception in service
Traceback (most recent call last):
  File "/Users/blue/yapapi/examples/utils/__init__.py", line 85, in run_golem_example
    loop.run_until_complete(task)
  File "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 629, in run_until_complete
    self.run_forever()
  File "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 596, in run_forever
    self._run_once()
  File "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 1854, in _run_once
    event_list = self._selector.select(timeout)
  File "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/selectors.py", line 562, in select
    kev_list = self._selector.control(None, max_ev, timeout)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/blue/.envs/yagna-python-tutorial/lib/python3.9/site-packages/yapapi/services/service_runner.py", line 256, in _run_instance
    batch = batch_task.result()
  File "/Users/blue/yapapi/examples/ssh/ssh.py", line 60, in start
    yield script
  File "/Users/blue/.envs/yagna-python-tutorial/lib/python3.9/site-packages/yapapi/services/service_runner.py", line 272, in _run_instance
    fut_result = yield batch
  File "/Users/blue/.envs/yagna-python-tutorial/lib/python3.9/site-packages/yapapi/engine.py", line 721, in process_batches
    results = await get_batch_results()
  File "/Users/blue/.envs/yagna-python-tutorial/lib/python3.9/site-packages/yapapi/engine.py", line 701, in get_batch_results
    async for event_class, event_kwargs in remote:
  File "/Users/blue/.envs/yagna-python-tutorial/lib/python3.9/site-packages/yapapi/rest/activity.py", line 263, in __aiter__
    raise BatchTimeoutError()
yapapi.rest.activity.BatchTimeoutError

ssh-yapapi-2022-03-15_13.25.04.log yagna_rCURRENT (4).log

[mfranciszkiewicz]

The issue is being closed, to be re-opened when necessary.

I have no longer encountered the issue since devnet providers have been restarted.

Before then, the investigation showed that the providers were operating under degraded performance conditions. The prevailing issue were the long database access times, which delayed each action taken by the provider agent (e.g. activity creation). Since the provider restart, further investigation was hampered and lead to examination of the following potential problems and symptoms:

• exhausted hardware resources

  There are multiple daemon and agent binaries running on each devnet machine. Resource utilization graphs in munin showed that the usage during the testing session wasn't out of ordinary

• dominant CPU usage by the hybrid net providers

  Hybrid devnet providers were suspectible of taking the most CPU power for challenge validation in P2P communication; requesting tasks simultaneously on the beta and hybrid devnets has proven this suspicion to be false

• hitting the open file descriptor limit

  This investigation lead to the fix in the erc20 driver where the HTTP client connection pool is re-used by all web3 calls: https://github.com/golemfactory/yagna/pull/1892 . The fix is mainly targeting the requestor nodes and will not impact provider nodes as much

Currently, provider nodes are behaving correctly.