search

raiden-network / raiden

Raiden Network
https://developer.raiden.network
Other
1.84k stars 378 forks source link

BF1 Fails due to Internal Server Error #5464

Closed agatsoh closed 4 years ago

agatsoh commented 4 years ago

Problem Definition

{
  "task": "<TransferTask: {'from': 3, 'to': 0, 'amount': 1000000000000000, 'lock_timeout': 30}>",
  "event": "Task errored",
  "logger": "scenario_player.tasks.base",
  "level": "error",
  "timestamp": "2019-12-08 03:39:43.419832",
  "exception": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.7/site-packages/scenario_player/tasks/base.py\", line 74, in __call__\n    return self._run(*args, **kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/scenario_player/tasks/api_base.py\", line 59, in _run\n    f'HTTP status code \"{resp.status_code}\" while fetching {url}. '\nscenario_player.exceptions.legacy.RESTAPIStatusMismatchError: HTTP status code \"500\" while fetching http://127.0.0.1:50441/api/v1/payments/0x62083c80353Df771426D209eF578619EE68D5C7A/0x09e9793f5D68AFFbd09eF2E671f8aA8bA0DFa48D. Expected 2..: Internal Server Error"
}

  1. Fails due to Internal Server Error Internal Server Error

  2. Fails at: - transfer: {from: 3, to: 0, amount: 1_000_000_000_000_000, lock_timeout: 30}

https://drive.google.com/drive/folders/12IDu6kI4Os5tpv4rOHkqIiy8xNWb69-t?usp=sharing

palango commented 4 years ago

This looks like a matrix problem:

--------- Starting ---------
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 716, in gevent._greenlet.Greenlet.run
  File "/usr/local/lib/python3.7/site-packages/raiden/network/transport/matrix/transport.py", line 267, in _run
    self._check_and_send()
  File "/usr/local/lib/python3.7/site-packages/raiden/network/transport/matrix/transport.py", line 251, in _check_and_send
    self.transport._send_raw(self.receiver, "\n".join(message_texts))
  File "/usr/local/lib/python3.7/site-packages/raiden/network/transport/matrix/transport.py", line 955, in _send_raw
    room.send_text(data)
  File "/usr/local/lib/python3.7/site-packages/matrix_client/room.py", line 105, in send_text
    return self.client.api.send_message(self.room_id, text)
  File "/usr/local/lib/python3.7/site-packages/matrix_client/api.py", line 336, in send_message
    timestamp=timestamp
  File "/usr/local/lib/python3.7/site-packages/matrix_client/api.py", line 256, in send_message_event
    return self._send("PUT", path, content, query_params=params)
  File "/usr/local/lib/python3.7/site-packages/raiden/network/transport/matrix/client.py", line 164, in _send
    return super()._send(method, path, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/matrix_client/api.py", line 691, in _send
    code=response.status_code, content=response.text
matrix_client.errors.MatrixRequestError: 404: 404 page not found

2019-12-08T03:39:42Z <Greenlet "RetryQueue node:0x382747B31Fa9cF94d0cF99273dfA302A41548348 recipient:0x7326f3b6271fEE9157c24851B28D373664e599f7" at 0x7f93f40df488: <bound method _RetryQueue._run of <_RetryQueue for 0x7326f3b6271fee9157c24851b28d373664e599f7>>> failed with MatrixRequestError

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/raiden/network/transport/matrix/transport.py", line 445, in _run
    self._broadcast_worker()
  File "/usr/local/lib/python3.7/site-packages/raiden/network/transport/matrix/transport.py", line 619, in _broadcast_worker
    self._broadcast_event.wait(self._config["retry_interval"])
  File "src/gevent/event.py", line 240, in gevent._event.Event.wait
  File "src/gevent/event.py", line 140, in gevent._event._AbstractLinkable._wait
  File "src/gevent/event.py", line 117, in gevent._event._AbstractLinkable._wait_core
  File "src/gevent/event.py", line 119, in gevent._event._AbstractLinkable._wait_core
  File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
matrix_client.errors.MatrixRequestError: 404: 404 page not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 716, in gevent._greenlet.Greenlet.run
  File "/usr/local/lib/python3.7/site-packages/raiden/network/transport/matrix/transport.py", line 452, in _run
    self.stop()  # ensure cleanup and wait on subtasks
  File "/usr/local/lib/python3.7/site-packages/raiden/network/transport/matrix/transport.py", line 481, in stop
    self._client.set_presence_state(UserPresence.OFFLINE.value)
  File "/usr/local/lib/python3.7/site-packages/raiden/network/transport/matrix/client.py", line 406, in set_presence_state
    "PUT", f"/presence/{quote(self.user_id)}/status", {"presence": state}
  File "/usr/local/lib/python3.7/site-packages/raiden/network/transport/matrix/client.py", line 164, in _send
    return super()._send(method, path, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/matrix_client/api.py", line 691, in _send
    code=response.status_code, content=response.text
matrix_client.errors.MatrixRequestError: 404: 404 page not found

2019-12-08T03:39:42Z <Greenlet "MatrixTransport._run node:0x382747B31Fa9cF94d0cF99273dfA302A41548348" at 0x7f93f5845d08: <bound method MatrixTransport._run of <MatrixTransport node:0x382747B31Fa9cF94d0cF99273dfA302A41548348 id:dc236edf-d674-4e65-99d9-5d92916f2118>>> failed with MatrixRequestError

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/raiden/network/transport/matrix/transport.py", line 445, in _run
    self._broadcast_worker()
  File "/usr/local/lib/python3.7/site-packages/raiden/network/transport/matrix/transport.py", line 619, in _broadcast_worker
    self._broadcast_event.wait(self._config["retry_interval"])
  File "src/gevent/event.py", line 240, in gevent._event.Event.wait
  File "src/gevent/event.py", line 140, in gevent._event._AbstractLinkable._wait
  File "src/gevent/event.py", line 117, in gevent._event._AbstractLinkable._wait_core
  File "src/gevent/event.py", line 119, in gevent._event._AbstractLinkable._wait_core
  File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
matrix_client.errors.MatrixRequestError: 404: 404 page not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 716, in gevent._greenlet.Greenlet.run
  File "/usr/local/lib/python3.7/site-packages/raiden/raiden_service.py", line 455, in _run
    self.stop_event.wait()
  File "src/gevent/event.py", line 240, in gevent._event.Event.wait
  File "src/gevent/event.py", line 140, in gevent._event._AbstractLinkable._wait
  File "src/gevent/event.py", line 117, in gevent._event._AbstractLinkable._wait_core
  File "src/gevent/event.py", line 119, in gevent._event._AbstractLinkable._wait_core
  File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
matrix_client.errors.MatrixRequestError: 404: 404 page not found

2019-12-08T03:39:42Z <Greenlet "RaidenService._run node:0x382747B31Fa9cF94d0cF99273dfA302A41548348" at 0x7f93f528d378: <bound method RaidenService._run of <RaidenService node:0x382747B31Fa9cF94d0cF99273dfA302A41548348>>> failed with MatrixRequestError

--------- Stopped ---------
Dominik1999 commented 4 years ago

Related to the crash of transport04 over the weekend. We might need a better error message / handling, see #5461