BF1 scenario SP fails in a wait task

[agatsoh]

Problem Definition

The SP fails in a wait task

{
  "task": "<WaitTask: 100>",
  "event": "Task errored",
  "logger": "scenario_player.tasks.base",
  "level": "error",
  "timestamp": "2019-11-28 05:07:36.190982",
  "exception": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.7/site-packages/scenario_player/tasks/base.py\", line 74, in __call__\n    return self._run(*args, **kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/scenario_player/tasks/execution.py\", line 72, in _run\n    gevent.sleep(self._config)\n  File \"/usr/local/lib/python3.7/site-packages/gevent/hub.py\", line 159, in sleep\n    hub.wait(t)\n  File \"src/gevent/_hub_primitives.py\", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait\n  File \"src/gevent/_hub_primitives.py\", line 55, in gevent.__hub_primitives.WaitOperationsGreenlet.wait\n  File \"src/gevent/_waiter.py\", line 151, in gevent.__waiter.Waiter.get\n  File \"src/gevent/_greenlet_primitives.py\", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch\n  File \"src/gevent/_greenlet_primitives.py\", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch\n  File \"src/gevent/_greenlet_primitives.py\", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch\n  File \"src/gevent/__greenlet_primitives.pxd\", line 35, in gevent.__greenlet_primitives._greenlet_switch\ngreenlet.GreenletExit"
}

What does the exception say:

1. Simplified traceback

   "Traceback (most recent call last):\n  
   File \"/usr/local/lib/python3.7/site-packages/scenario_player/tasks/base.py\", line 74, in __call__\n    return self._run(*args, **kwargs)\n  
   File \"/usr/local/lib/python3.7/site-packages/scenario_player/tasks/execution.py\", line 72, in _run\n    gevent.sleep(self._config)\n  
   File \"/usr/local/lib/python3.7/site-packages/gevent/hub.py\", line 159, in sleep\n    hub.wait(t)\n  
   File \"src/gevent/_hub_primitives.py\", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait\n  
   File \"src/gevent/_hub_primitives.py\", line 55, in gevent.__hub_primitives.WaitOperationsGreenlet.wait\n  
   File \"src/gevent/_waiter.py\", line 151, in gevent.__waiter.Waiter.get\n  
   File \"src/gevent/_greenlet_primitives.py\", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch\n  
   File \"src/gevent/_greenlet_primitives.py\", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch\n  
   File \"src/gevent/_greenlet_primitives.py\", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch\n  
   File \"src/gevent/__greenlet_primitives.pxd\", line 35, in gevent.__greenlet_primitives._greenlet_switch\ngreenlet.GreenletExit"

2. Fails while waiting for task the next exception gives the exact task

{
  "task": "<SerialTask: {'name': 'Stop node 0 and wait 100s, then start it again', 'tasks': [{'stop_node': 0}, {'wait': 100}, {'start_node': 0}, {'wait': 100}]}>",
  "event": "Task errored",
  "logger": "scenario_player.tasks.base",
  "level": "error",
  "timestamp": "2019-11-28 05:07:36.193097",
  "exception": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.7/site-packages/scenario_player/tasks/base.py\", line 74, in __call__\n    return self._run(*args, **kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/scenario_player/tasks/execution.py\", line 41, in _run\n    task()\n  File \"/usr/local/lib/python3.7/site-packages/scenario_player/tasks/base.py\", line 74, in __call__\n    return self._run(*args, **kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/scenario_player/tasks/execution.py\", line 72, in _run\n    gevent.sleep(self._config)\n  File \"/usr/local/lib/python3.7/site-packages/gevent/hub.py\", line 159, in sleep\n    hub.wait(t)\n  File \"src/gevent/_hub_primitives.py\", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait\n  File \"src/gevent/_hub_primitives.py\", line 55, in gevent.__hub_primitives.WaitOperationsGreenlet.wait\n  File \"src/gevent/_waiter.py\", line 151, in gevent.__waiter.Waiter.get\n  File \"src/gevent/_greenlet_primitives.py\", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch\n  File \"src/gevent/_greenlet_primitives.py\", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch\n  File \"src/gevent/_greenlet_primitives.py\", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch\n  File \"src/gevent/__greenlet_primitives.pxd\", line 35, in gevent.__greenlet_primitives._greenlet_switch\ngreenlet.GreenletExit"
}

One of the waits in Stop node 0 and wait 100s, then start it again

https://drive.google.com/file/d/1QNntSD9h_7mlWxEFdamlQorHoOaKRG22/view?usp=sharing

[agatsoh]

This issue also may be related to https://github.com/raiden-network/scenario-player/issues/433

[palango]

The logs don't include the reason for the exception. The error might be written to stderr or there is some other reason (OOM) for the scenario to be terminated.

[agatsoh]

The BF1 scenario fails again today at the almost the same place but during the stop node task

{
  "task": "<StopNodeTask: 0>",
  "event": "Task errored",
  "logger": "scenario_player.tasks.base",
  "level": "error",
  "timestamp": "2019-11-29 05:09:58.920911",
  "exception": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.7/site-packages/scenario_player/tasks/base.py\", line 74, in __call__\n    return self._run(*args, **kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/scenario_player/tasks/raiden_node.py\", line 16, in _run\n    method()\n  File \"/usr/local/lib/python3.7/site-packages/scenario_player/node_support.py\", line 229, in stop\n    ret = self.executor.stop(timeout=timeout)\n  File \"/usr/local/lib/python3.7/site-packages/scenario_player/utils/legacy.py\", line 170, in stop\n    self.wait_for(process_stopped)\n  File \"/usr/local/lib/python3.7/site-packages/mirakuru/base.py\", line 350, in wait_for\n    time.sleep(self._sleep)\n  File \"/usr/local/lib/python3.7/site-packages/gevent/hub.py\", line 159, in sleep\n    hub.wait(t)\n  File \"src/gevent/_hub_primitives.py\", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait\n  File \"src/gevent/_hub_primitives.py\", line 55, in gevent.__hub_primitives.WaitOperationsGreenlet.wait\n  File \"src/gevent/_waiter.py\", line 151, in gevent.__waiter.Waiter.get\n  File \"src/gevent/_greenlet_primitives.py\", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch\n  File \"src/gevent/_greenlet_primitives.py\", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch\n  File \"src/gevent/_greenlet_primitives.py\", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch\n  File \"src/gevent/__greenlet_primitives.pxd\", line 35, in gevent.__greenlet_primitives._greenlet_switch\ngreenlet.GreenletExit"
}

bf1_basic_functionality_2019_11_29.tar.gz

[agatsoh]

On30thNovember2019 the PFS8 Similarly fails in a wait task.

{"task": "<WaitBlocksTask: 1>", "event": "Task errored", "logger": "scenario_player.tasks.base", "level": "error", "timestamp": "2019-11-30 03:47:39.931497", "exception": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.7/site-packages/scenario_player/tasks/base.py\", line 74, in __call__\n    return self._run(*args, **kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/scenario_player/tasks/execution.py\", line 84, in _run\n    gevent.sleep(10)\n  File \"/usr/local/lib/python3.7/site-packages/gevent/hub.py\", line 159, in sleep\n    hub.wait(t)\n  File \"src/gevent/_hub_primitives.py\", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait\n  File \"src/gevent/_hub_primitives.py\", line 55, in gevent.__hub_primitives.WaitOperationsGreenlet.wait\n  File \"src/gevent/_waiter.py\", line 151, in gevent.__waiter.Waiter.get\n  File \"src/gevent/_greenlet_primitives.py\", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch\n  File \"src/gevent/_greenlet_primitives.py\", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch\n  File \"src/gevent/_greenlet_primitives.py\", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch\n  File \"src/gevent/__greenlet_primitives.pxd\", line 35, in gevent.__greenlet_primitives._greenlet_switch\ngreenlet.GreenletExit"}

https://drive.google.com/drive/folders/1WkIkO-naWPFaWA97nGg1BNa1us8Jr10-?usp=sharing

[agatsoh]

On 2ndDecember2019 PFS again fails on a wait task https://drive.google.com/drive/folders/1nn48zGfEIuEd4K5bkzD23yiWfusdtkcC?usp=sharing

[agatsoh]

BF1 scenarios errors for both 30th september and 2nd December are both for wait task errors

[agatsoh]

On 5thDecember2019 pfs8 fails on wait task:

{
  "task": "<WaitBlocksTask: 1>",
  "event": "Task errored",
  "logger": "scenario_player.tasks.base",
  "level": "error",
  "timestamp": "2019-12-05 03:52:33.653052",
  "exception": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.7/site-packages/scenario_player/tasks/base.py\", line 74, in __call__\n    return self._run(*args, **kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/scenario_player/tasks/execution.py\", line 83, in _run\n    while web3.eth.blockNumber < end_block:\n  File \"/usr/local/lib/python3.7/site-packages/web3/eth.py\", line 102, in blockNumber\n    return self.web3.manager.request_blocking(\"eth_blockNumber\", [])\n  File \"/usr/local/lib/python3.7/site-packages/web3/manager.py\", line 109, in request_blocking\n    response = self._make_request(method, params)\n  File \"/usr/local/lib/python3.7/site-packages/web3/manager.py\", line 92, in _make_request\n    return request_func(method, params)\n  File \"/usr/local/lib/python3.7/site-packages/web3/middleware/cache.py\", line 105, in middleware\n    return make_request(method, params)\n  File \"cytoolz/functoolz.pyx\", line 232, in cytoolz.functoolz.curry.__call__\n  File \"/usr/local/lib/python3.7/site-packages/web3/middleware/formatting.py\", line 50, in apply_formatters\n    response = make_request(method, params)\n  File \"/usr/local/lib/python3.7/site-packages/web3/middleware/gas_price_strategy.py\", line 18, in middleware\n    return make_request(method, params)\n  File \"cytoolz/functoolz.pyx\", line 232, in cytoolz.functoolz.curry.__call__\n  File \"/usr/local/lib/python3.7/site-packages/web3/middleware/formatting.py\", line 50, in apply_formatters\n    response = make_request(method, params)\n  File \"/usr/local/lib/python3.7/site-packages/web3/middleware/attrdict.py\", line 18, in middleware\n    response = make_request(method, params)\n  File \"cytoolz/functoolz.pyx\", line 232, in cytoolz.functoolz.curry.__call__\n  File \"/usr/local/lib/python3.7/site-packages/web3/middleware/formatting.py\", line 50, in apply_formatters\n    response = make_request(method, params)\n  File \"/usr/local/lib/python3.7/site-packages/web3/middleware/normalize_errors.py\", line 9, in middleware\n    result = make_request(method, params)\n  File \"cytoolz/functoolz.pyx\", line 232, in cytoolz.functoolz.curry.__call__\n  File \"/usr/local/lib/python3.7/site-packages/web3/middleware/formatting.py\", line 50, in apply_formatters\n    response = make_request(method, params)\n  File \"cytoolz/functoolz.pyx\", line 232, in cytoolz.functoolz.curry.__call__\n  File \"/usr/local/lib/python3.7/site-packages/web3/middleware/formatting.py\", line 50, in apply_formatters\n    response = make_request(method, params)\n  File \"cytoolz/functoolz.pyx\", line 232, in cytoolz.functoolz.curry.__call__\n  File \"/usr/local/lib/python3.7/site-packages/web3/middleware/formatting.py\", line 50, in apply_formatters\n    response = make_request(method, params)\n  File \"/usr/local/lib/python3.7/site-packages/web3/middleware/exception_retry_request.py\", line 80, in middleware\n    return make_request(method, params)\n  File \"/usr/local/lib/python3.7/site-packages/web3/providers/rpc.py\", line 68, in make_request\n    **self.get_request_kwargs()\n  File \"/usr/local/lib/python3.7/site-packages/web3/utils/request.py\", line 26, in make_post_request\n    response = session.post(endpoint_uri, data=data, *args, **kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/requests/sessions.py\", line 581, in post\n    return self.request('POST', url, data=data, json=json, **kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/requests/sessions.py\", line 533, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/requests/sessions.py\", line 646, in send\n    r = adapter.send(request, **kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/requests/adapters.py\", line 449, in send\n    timeout=timeout\n  File \"/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 603, in urlopen\n    chunked=chunked)\n  File \"/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 383, in _make_request\n    httplib_response = conn.getresponse()\n  File \"/usr/local/lib/python3.7/http/client.py\", line 1321, in getresponse\n    response.begin()\n  File \"/usr/local/lib/python3.7/http/client.py\", line 296, in begin\n    version, status, reason = self._read_status()\n  File \"/usr/local/lib/python3.7/http/client.py\", line 257, in _read_status\n    line = str(self.fp.readline(_MAXLINE + 1), \"iso-8859-1\")\n  File \"/usr/local/lib/python3.7/socket.py\", line 589, in readinto\n    return self._sock.recv_into(b)\n  File \"/usr/local/lib/python3.7/site-packages/gevent/_socket3.py\", line 433, in recv_into\n    self._wait(self._read_event)\n  File \"src/gevent/_hub_primitives.py\", line 265, in gevent.__hub_primitives.wait_on_socket\n  File \"src/gevent/_hub_primitives.py\", line 266, in gevent.__hub_primitives.wait_on_socket\n  File \"src/gevent/_hub_primitives.py\", line 261, in gevent.__hub_primitives._primitive_wait\n  File \"src/gevent/_hub_primitives.py\", line 262, in gevent.__hub_primitives._primitive_wait\n  File \"src/gevent/_hub_primitives.py\", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait\n  File \"src/gevent/_hub_primitives.py\", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait\n  File \"src/gevent/_hub_primitives.py\", line 55, in gevent.__hub_primitives.WaitOperationsGreenlet.wait\n  File \"src/gevent/_waiter.py\", line 151, in gevent.__waiter.Waiter.get\n  File \"src/gevent/_greenlet_primitives.py\", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch\n  File \"src/gevent/_greenlet_primitives.py\", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch\n  File \"src/gevent/_greenlet_primitives.py\", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch\n  File \"src/gevent/__greenlet_primitives.pxd\", line 35, in gevent.__greenlet_primitives._greenlet_switch\ngreenlet.GreenletExit"
}

pfs8_mediator_goes_offline_2019_12_05.tar.gz