MatrixRequestError 500 after some time when idle

[vanous]

i have long running bot made of the matrix python client and in past couple of weeks, it would randomly (often overnight) stop receiving incoming traffic with this error message (sorry, i have screenshot only):

The error is too deep in threading for me to understand :/ , sorry. Interestingly enough, this started repeatedly about week ago , this last night it didn't happen.... I am reachable @vanous:matrix.org . thank you

[erikjohnston]

Is this against the matrix.org server? If so, what's the user ID of the bot? Looks like there may be a server side bug, given MatrixRequestError: 500, but the SDK should retry on 500's.

[vanous]

yes, matrix HS, by ID i presume you mean username, it is "podoli-server".

[erikjohnston]

Thanks, there don't seem to be any reported errors, so it may just be that we restarted synapse at that time.

So the bug is simply that the SDK needs to retry requests on 5xx responses from the server.

[vanous]

Before my report, i had tried to sandwich some return True around in the listen_forever / start_listener_thread but that didn't help.... but my knowledge is unfortunately limited...

[erikjohnston]

I think @Half-Shot is currently looking at rewriting that bit of code atm, which will probably fix this.

[Half-Shot]

Closing this as I believe it is fixed. Feel free to reopen @vanous if you run into issues again.

[vanous]

huh, i am too late here :) ...thanks for the release, i just returned from short vacation and it seems working.

One comment about the pypi package, might as well put it here, feel free to move some place else:

pip still installs 0.0.3, not sure why...:

pip install -vvv matrix-client Converted retries value: Retry(total=5, connect=None, read=None, redirect=None) -> Retry(total=Retry(total=5, connect=None, read=None, redirect=None), connect=None, read=None, redirect=None) Converted retries value: Retry(total=5, connect=None, read=None, redirect=None) -> Retry(total=Retry(total=5, connect=None, read=None, redirect=None), connect=None, read=None, redirect=None) Collecting matrix-client 1 location(s) to search for versions of matrix-client:

• https://pypi.python.org/simple/matrix-client/ Getting page https://pypi.python.org/simple/matrix-client/ Analyzing links from page https://pypi.python.org/simple/matrix-client/ Found link https://pypi.python.org/packages/0c/81/36e6fbddfa34c9507e1387ec62b8539589999cd2726c8199dcc7653c1213/matrix-client-0.0.3.tar.gz#md5=4f392e3f9b8b564cbd7a3fe4fdd36577 (from https://pypi.python.org/simple/matrix-client/), version: 0.0.3 Found link https://pypi.python.org/packages/8a/d4/3c6d2e8cf894126f623b23e5cdebfe530980e80e65212a362b560c71e417/matrix_client-0.0.1.tar.gz#md5=a80f19752fada7cebc064b898cd09791 (from https://pypi.python.org/simple/matrix-client/), version: 0.0.1 Found link https://pypi.python.org/packages/ce/59/c9d91f9df2f52c6a54a90ad15c77b26b6edaaf21510259d130c78ee41384/matrix_client-0.0.2.tar.gz#md5=569bd72f26c3e89202510176862e4757 (from https://pypi.python.org/simple/matrix-client/), version: 0.0.2 Using version 0.0.3 (newest of versions: 0.0.1, 0.0.2, 0.0.3) Using cached wheel link: file:///home/vanous/.cache/pip/wheels/58/ac/43/29796ca208da97e9e3dd4ce117c440068fdd4dcb254062c010/matrix_client-0.0.3-cp27-none-any.whl Requirement already satisfied (use --upgrade to upgrade): requests in ./lib/python2.7/site-packages (from matrix-client) Installing collected packages: matrix-client

Successfully installed matrix-client-0.0.3 Cleaning up... Converted retries value: Retry(total=0, connect=None, read=None, redirect=None) -> Retry(total=Retry(total=0, connect=None, read=None, redirect=None), connect=None, read=None, redirect=None) Converted retries value: Retry(total=0, connect=None, read=None, redirect=None) -> Retry(total=Retry(total=0, connect=None, read=None, redirect=None), connect=None, read=None, redirect=None)

[Half-Shot]

https://pypi.python.org/pypi/matrix-client/0.0.4 :/ That's...very strange.

EDIT: It does indeed. I'll investigate.

[vanous]

yes. Tried on two computers, erased caches in .cache/pip to no avail... i then installed it correctly via pip install url.

[vanous]

Perhaps this can help... PyPI Release Checklist: https://gist.github.com/audreyr/5990987

[vanous]

still on topic, but probably not an issue, just FYI. I have tested and if i long term heavily over-load my internet connection, i can get connection issues. So here is example how the new sync performs under these conditions. feel free to ignore or let me know if i can be of any help testing it. cheers P.

debug.txt

[Half-Shot]

(Fixed the PyPI issue btw, I did something extremely daft and it's all fine now)

Thanks for the log, Looks like we are still running into issues with the spam. Does the bot continue to work after these errors?

[vanous]

yes, the bot still works, see another debug file. Sometimes i have to submit my command few times before connection is re-established, then it works.

In the log, these lines:

msg received: @vanous !debug msg received: @vanous !address msg received: @vanous !webcam msg received: @vanous !debug msg received: @vanous !debug DEBUG

are my commands to the bot. (I can now change matrix-client logging level during runtime, to have more or less verbosity)

deb.txt

i have again removed token, but kept room id.

Edit: the blocked network is surely caused by buffer bloat. Way to test was to start downloading linux client for WarThunder for the teenager in my house :)

Edit 2: it wouldn't be a bad idea to have timestamp with the log entires.

[vanous]

Btw, still can see the 500 error, this happens during start/login. Few tries and then it works.

Traceback (most recent call last): File "reader.py", line 26, in token=client.login_with_password(my_username,my_password) File "/usr/local/lib/python2.7/dist-packages/matrix_client/client.py", line 134, in login_with_password "m.login.password", user=username, password=password File "/usr/local/lib/python2.7/dist-packages/matrix_client/api.py", line 144, in login return self._send("POST", "/login", content) File "/usr/local/lib/python2.7/dist-packages/matrix_client/api.py", line 400, in _send code=response.status_code, content=response.text matrix_client.api.MatrixRequestError: 500: pi@raspberrypi:~/bin $ pi@raspberrypi:~/bin $ python reader.py Traceback (most recent call last): File "reader.py", line 26, in token=client.login_with_password(my_username,my_password) File "/usr/local/lib/python2.7/dist-packages/matrix_client/client.py", line 134, in login_with_password "m.login.password", user=username, password=password File "/usr/local/lib/python2.7/dist-packages/matrix_client/api.py", line 144, in login return self._send("POST", "/login", content) File "/usr/local/lib/python2.7/dist-packages/matrix_client/api.py", line 400, in _send code=response.status_code, content=response.text matrix_client.api.MatrixRequestError: 500: pi@raspberrypi:~/bin $ python reader.py Traceback (most recent call last): File "reader.py", line 26, in token=client.login_with_password(my_username,my_password) File "/usr/local/lib/python2.7/dist-packages/matrix_client/client.py", line 134, in login_with_password "m.login.password", user=username, password=password File "/usr/local/lib/python2.7/dist-packages/matrix_client/api.py", line 144, in login return self._send("POST", "/login", content) File "/usr/local/lib/python2.7/dist-packages/matrix_client/api.py", line 400, in _send code=response.status_code, content=response.text matrix_client.api.MatrixRequestError: 500:

[Half-Shot]

500s are odd. That's usually a server problem :/