Closed vorpal-buildbot closed 7 years ago
bakert
commented
7 years ago See also #1023 #1024 #1025 #1026 #1027 #1028 #1029 #1030 #1031 #1032 #1033 #1035
Some kind of intermittent network problem (?) over the last eight hours. Or we're doing something different?
bakert
commented
7 years ago Storing http://magic.bluebones.net/proxies/?c=exava%2c+rakdos+blood+witch%7clightning+mauler%7creclusive+artificer%7csurrak%2c+the+hunt+caller%7cthopter+engineer%7cakoum+battlesinger%7csamut%2c+voice+of+dissent in images/exava--rakdos-blood-witch_lightning-mauler_reclusive-artificer_surrak--the-hunt-caller_thopter-engineer_akoum-battlesinger_samut--voice-of-dissent.jpg File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/bot.py", line 119, in on_message
await BOT.on_message(message)
File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/bot.py", line 32, in on_message
await self.respond_to_card_names(message)
File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/bot.py", line 44, in respond_to_card_names
await command.respond_to_card_names(message, self)
File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/command.py", line 24, in respond_to_card_names
await bot.post_cards(cards, message.channel, message.author)
File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/bot.py", line 98, in post_cards
message = await self.client.send_file(channel, image_file, content=text)
File "/home/discord/.local/lib/python3.5/site-packages/discord/client.py", line 1235, in send_file
filename=filename, content=content, tts=tts)
File "/home/discord/.local/lib/python3.5/site-packages/discord/http.py", line 137, in request
r = yield from self.session.request(method, url, **kwargs)
File "/home/discord/.local/lib/python3.5/site-packages/aiohttp/client.py", line 555, in __iter__
resp = yield from self._coro
File "/home/discord/.local/lib/python3.5/site-packages/aiohttp/client.py", line 209, in _request
raise aiohttp.ClientResponseError() from exc
aiohttp.errors.ClientResponseError
Ignoring exception in on_message
Traceback (most recent call last):
File "/usr/lib64/python3.5/asyncio/selector_events.py", line 664, in _read_ready
data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/discord/.local/lib/python3.5/site-packages/aiohttp/client.py", line 202, in _request
yield from resp.start(conn, read_until_eof)
File "/home/discord/.local/lib/python3.5/site-packages/aiohttp/client_reqrep.py", line 640, in start
message = yield from httpstream.read()
File "/home/discord/.local/lib/python3.5/site-packages/aiohttp/streams.py", line 641, in read
result = yield from super().read()
File "/home/discord/.local/lib/python3.5/site-packages/aiohttp/streams.py", line 476, in read
yield from self._waiter
File "/usr/lib64/python3.5/asyncio/futures.py", line 358, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib64/python3.5/asyncio/tasks.py", line 290, in _wakeup
future.result()
File "/usr/lib64/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
aiohttp.errors.ServerDisconnectedError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/discord/.local/lib/python3.5/site-packages/discord/client.py", line 307, in _run_event
yield from getattr(self, event)(*args, **kwargs)
File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/bot.py", line 119, in on_message
await BOT.on_message(message)
File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/bot.py", line 32, in on_message
await self.respond_to_card_names(message)
File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/bot.py", line 44, in respond_to_card_names
await command.respond_to_card_names(message, self)
File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/command.py", line 24, in respond_to_card_names
await bot.post_cards(cards, message.channel, message.author)
File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/bot.py", line 98, in post_cards
message = await self.client.send_file(channel, image_file, content=text)
File "/home/discord/.local/lib/python3.5/site-packages/discord/client.py", line 1235, in send_file
filename=filename, content=content, tts=tts)
File "/home/discord/.local/lib/python3.5/site-packages/discord/http.py", line 137, in request
r = yield from self.session.request(method, url, **kwargs)
File "/home/discord/.local/lib/python3.5/site-packages/aiohttp/client.py", line 555, in __iter__
resp = yield from self._coro
File "/home/discord/.local/lib/python3.5/site-packages/aiohttp/client.py", line 209, in _request
raise aiohttp.ClientResponseError() from exc
aiohttp.errors.ClientResponseError
Ignoring exception in on_message
Traceback (most recent call last):
File "/usr/lib64/python3.5/asyncio/selector_events.py", line 664, in _read_ready
data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/discord/.local/lib/python3.5/site-packages/aiohttp/client.py", line 202, in _request
yield from resp.start(conn, read_until_eof)
File "/home/discord/.local/lib/python3.5/site-packages/aiohttp/client_reqrep.py", line 640, in start
message = yield from httpstream.read()
File "/home/discord/.local/lib/python3.5/site-packages/aiohttp/streams.py", line 641, in read
result = yield from super().read()
File "/home/discord/.local/lib/python3.5/site-packages/aiohttp/streams.py", line 476, in read
yield from self._waiter
File "/usr/lib64/python3.5/asyncio/futures.py", line 358, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib64/python3.5/asyncio/tasks.py", line 290, in _wakeup
future.result()
File "/usr/lib64/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
aiohttp.errors.ServerDisconnectedError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/discord/.local/lib/python3.5/site-packages/discord/client.py", line 307, in _run_event
yield from getattr(self, event)(*args, **kwargs)
File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/bot.py", line 119, in on_message
await BOT.on_message(message)
File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/bot.py", line 32, in on_message
await self.respond_to_card_names(message)
File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/bot.py", line 44, in respond_to_card_names
await command.respond_to_card_names(message, self)
File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/command.py", line 24, in respond_to_card_names
await bot.post_cards(cards, message.channel, message.author)
File "/home/discord/Penny-Dreadful-Discord-Bot/discordbot/bot.py", line 98, in post_cards
message = await self.client.send_file(channel, image_file, content=text)
File "/home/discord/.local/lib/python3.5/site-packages/discord/client.py", line 1235, in send_file
filename=filename, content=content, tts=tts)
File "/home/discord/.local/lib/python3.5/site-packages/discord/http.py", line 137, in request
r = yield from self.session.request(method, url, **kwargs)
File "/home/discord/.local/lib/python3.5/site-packages/aiohttp/client.py", line 555, in __iter__
resp = yield from self._coro
File "/home/discord/.local/lib/python3.5/site-packages/aiohttp/client.py", line 209, in _request
raise aiohttp.ClientResponseError() from exc
aiohttp.errors.ClientResponseError
Ignoring exception in on_message
Traceback (most recent call last):
File "/usr/lib64/python3.5/asyncio/selector_events.py", line 664, in _read_ready
data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/discord/.local/lib/python3.5/site-packages/aiohttp/client.py", line 202, in _request
yield from resp.start(conn, read_until_eof)
File "/home/discord/.local/lib/python3.5/site-packages/aiohttp/client_reqrep.py", line 640, in start
message = yield from httpstream.read()
File "/home/discord/.local/lib/python3.5/site-packages/aiohttp/streams.py", line 641, in read
result = yield from super().read()
File "/home/discord/.local/lib/python3.5/site-packages/aiohttp/streams.py", line 476, in read
yield from self._waiter
File "/usr/lib64/python3.5/asyncio/futures.py", line 358, in __iter__
yield self # This tells Task to wait for completion.So all of these happened over a 7 hour period 21-14 hours ago. It might be "bot overloading" from the rotation. That seems more likely than Discord randomly having issues at our busiest time.
I'm not sure what the amelioration should be in that case. Catch ClientResponseError or ServerDisconnectedError and say "I'm really busy right now". If we even can? Start to limit image sending or something similar for a few minutes after catching one of these?
silasary
commented
7 years ago Aggressively cache /rotation/?
bakert
commented
7 years ago Still happening: #1061
bakert
commented
7 years ago One possibility here is to add some retrying logic. Assuming this is genuine "we did so much discord flipped out" and not "we changed something and broke the bot".
silasary
commented
7 years ago It might also be the fact that the server is going silly with CPU usage.
bakert
commented
7 years ago One suspicion is that execute the complex SQL in price.cache is CPU-intensive enough that it causes the whole server to have issues. It certainly does use a lot of CPU for many seconds/even minutes and should be made more efficient or done some other way.
bakert
commented
7 years ago 8 more tasks like this have been logged over the last two days. And I think a lot of ones that would be logged are not making it to us because the bot does not respond at all. I've closed them and leaving this one open. Working on the price SQL as our best guess.
bakert
commented
7 years ago Pretty sure this was price grabber which now performs much better. Closing this but the bot will log a new issue if it happens more.
Reported on Discord by Back_Alley_G#3571