Open dikderoy opened 7 years ago
according to this tread: http://stackoverflow.com/questions/6650940/interfaceerror-0 this is related to unreleased cursors somehow.
here I must mention what I use Manager.atomic()
context manager, maybe there is a problem in releasing cursors due to following statement:
@asyncio.coroutine
def cursor(self, conn=None, *args, **kwargs):
"""Get cursor for connection from pool.
"""
in_transaction = conn is not None
if not conn:
conn = yield from self.acquire()
cursor = yield from conn.cursor(*args, **kwargs)
# NOTE: `cursor.release` is an awaitable object!
cursor.release = self.release_cursor(
cursor, in_transaction=in_transaction)
return cursor
@asyncio.coroutine
def release_cursor(self, cursor, in_transaction=False):
"""Release cursor coroutine. Unless in transaction,
the connection is also released back to the pool.
"""
conn = cursor.connection
yield from cursor.close()
if not in_transaction:
self.release(conn)
do context manager releases cursors after exiting a transaction?
Hello @dikderoy! Could you provide complete code snippet to reproduce the issue?
here I must mention what I use Manager.atomic() context manager
Transactions in async mode is a kind of tricky thing. Usually in blocking mode connection for transaction is saved into thread locals so it can't interfere with other connections from other threads / transactions.
Is sync mode we can't use thread locals, so we're using per-task context. That means, each transactions should run within separate async task. So, I'm not sure, It may be some edge case.
it occurs on this code (quite tricky, nested coroutines and several atomics()
)
async def update(self, entity: DomainItem, existing: Item = None) -> bool:
async with self._manager.atomic() as trx:
existing = existing or False
if not existing:
candidates = await self._manager.prefetch(
Item.select().where(Item.id == entity.cid),
Item2Source.select().where(Item2Source.item_id == entity.cid)
)
for one in candidates: # type: Item
if one.id == entity.cid:
existing = one # type: Item
break
if existing: # type: Item
# check instance version matches (optimistic lock)
if not existing.version == entity.data.version:
raise VersionMismatch()
entity.data.version += 1
# update peewee instance with data from domain instance (sync)
DomainEncoder.item_update_instance(existing, entity)
await self._manager.update(existing)
# merging source references
# nested atomic in same context
async with self._manager.atomic():
if entity.meta.sid not in [ref.source_id for ref in existing.item2source_set_prefetch]:
source = await self._manager.get(Source, Source.id == entity.meta.sid) # type: Source
await self._manager.create(Item2Source, item_id=existing.id, source_id=source.id)
return True
return False
async def save(self, entity: DomainItem) -> bool:
async with self._manager.atomic() as trx:
try:
existing = False
presaved = await self._manager.prefetch(
Item.select().where(Item.id == entity.cid),
Item2Source.select().where(Item2Source.item_id == entity.cid)
)
for one in presaved: # type: Item
if one.id == entity.cid:
existing = one
break
if existing:
return await self.update(entity, existing) # coro with nested atomic context
else:
db_model = await self._manager.create(Item, **DomainEncoder.item_encode_dict(entity))
await trx.commit() # needed to get ID (or am I doing something wrong?)
entity.cid = db_model.id
db_rel = await self._manager.create(Item2Source, source_id=entity.meta.sid, item_id=db_model.id)
await trx.commit() # needed to get ID (or am I doing something wrong?)
entity.meta.sid = db_rel.source_id
if len(entity.data.images) > 0:
for image in entity.data.images:
await self._manager.create(ItemImage, **DomainEncoder.image_encode_dict(image),
item_id=db_model.id)
return True
except peewee.PeeweeException as e:
raise StorageException(e)
later I will try to write an app to reproduce the issue, for now this is a snippet from my project itself as is. I've commented some parts so You may understand what's going on around... hope it gives you a lead.
according to stacktrace - it appears in case then
update
is called from within the create
and inside of nested atomic()
at this line:
27: await self._manager.create(Item2Source, item_id=existing.id, source_id=source.id)
at 3rd "level"
looks like the issue has nothing to do with atomic
stack..
i've refactored my code, removing nested atomics, and make the problematic call with no transaction at all, problem persists..
and after relaunch - it works normally...
so far my investigation led me to following:
connection
instance for more than one asyncio.Task
I forged a simple app with a unit test which reproduce an issue.
resulting exception is different (not InterfaceError) but I think these two cases are connected.
here is a repo with an app - if you're interested: https://github.com/dikderoy/peewee-async-bug
and here is a trace:
Error
Traceback (most recent call last):
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/peewee.py", line 3748, in execute_sql
cursor.execute(sql, params or ())
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/pymysql/cursors.py", line 166, in execute
result = self._query(query)
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/pymysql/cursors.py", line 322, in _query
conn.query(q)
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/pymysql/connections.py", line 852, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/pymysql/connections.py", line 1053, in _read_query_result
result.read()
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/pymysql/connections.py", line 1336, in read
first_packet = self.connection._read_packet()
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/pymysql/connections.py", line 983, in _read_packet
packet_header = self._read_bytes(4)
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/pymysql/connections.py", line 1029, in _read_bytes
CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/deroy/PROJECTS/Pets/peewee-async-bug/app/test_item_save_handler.py", line 60, in test_2_fails
asyncio.get_event_loop().run_until_complete(test())
File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
return future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "/home/deroy/PROJECTS/Pets/peewee-async-bug/app/test_item_save_handler.py", line 57, in test
await t
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
result = coro.send(None)
File "/home/deroy/PROJECTS/Pets/peewee-async-bug/app/test_item_save_handler.py", line 67, in add_record
item = await irepo.save({'id': identifier, 'sid': 1, 'title': 'i_%s' % identifier})
File "/home/deroy/PROJECTS/Pets/peewee-async-bug/app/sql.py", line 138, in save
db_rel = await self._manager.create(Item2Source, source_id=entity['sid'], item=db_model)
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/peewee_async.py", line 183, in create
pk = inst._get_pk_value()
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/peewee.py", line 5017, in _get_pk_value
return getattr(self, self._meta.primary_key.name)
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/peewee.py", line 1553, in __get__
for field_name in self.field_names])
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/peewee.py", line 1553, in <listcomp>
for field_name in self.field_names])
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/peewee.py", line 1363, in __get__
return self.get_object_or_id(instance)
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/peewee.py", line 1354, in get_object_or_id
obj = self.rel_model.get(self.field.to_field == rel_id)
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/peewee.py", line 4900, in get
return sq.get()
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/peewee.py", line 3161, in get
return next(clone.execute())
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/peewee.py", line 3213, in execute
self._qr = ResultWrapper(model_class, self._execute(), query_meta)
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/peewee.py", line 2892, in _execute
return self.database.execute_sql(sql, params, self.require_commit)
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/peewee_async.py", line 1024, in execute_sql
return super().execute_sql(*args, **kwargs)
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/peewee.py", line 3755, in execute_sql
self.commit()
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/peewee.py", line 3578, in __exit__
reraise(new_type, new_type(*exc_args), traceback)
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/peewee.py", line 135, in reraise
raise value.with_traceback(tb)
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/peewee.py", line 3748, in execute_sql
cursor.execute(sql, params or ())
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/pymysql/cursors.py", line 166, in execute
result = self._query(query)
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/pymysql/cursors.py", line 322, in _query
conn.query(q)
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/pymysql/connections.py", line 852, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/pymysql/connections.py", line 1053, in _read_query_result
result.read()
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/pymysql/connections.py", line 1336, in read
first_packet = self.connection._read_packet()
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/pymysql/connections.py", line 983, in _read_packet
packet_header = self._read_bytes(4)
File "/home/deroy/PROJECTS/venv/lib/python3.5/site-packages/pymysql/connections.py", line 1029, in _read_bytes
CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
peewee.OperationalError: (2013, 'Lost connection to MySQL server during query')
Hi @dikderoy!
I've performed a little investigation, here are some thoughts.
I can reproduce the issue just restarting MySQL server during sleep()
call, so we don't need to wait 20 minutes, 30 seconds is enough: start tests with python -m app.test_item_save_handler
and restart server after add 3rd record
message.
The original exception as I can see comes from sync query, peewee.execute_sql()
is a sync one. So, the reason of exception is not necessary in async code.
But what bothers me is that sync query is triggered, I think I'll dive into it some time later.
my test is designed to show what bug appears in simple, real-world scenario, "crazy-monkey method" to speed up tests - is just a way of getting work done faster =)
Here is another thing which can help to track the cause - in all variations of code and execution graphs - it always fail while working with Item2Source
- which is not like other 2 models - it is a many2many table relation (composed of 2 foreignkey fields and having them as composite pk)
I am not familiar with peewee internals but it looks connected.
Experiencing the same InterfaceError, any progress on this?
I am also encountering this. I use flask with socketio, which, as far as I know, uses coroutines or something similar. Maybe it is connected to async, after all?
I am not using anything fancy from peewee - just the high level API. And indeed, it happens after long idle time.
Edit: Looks like I got the wrong repo, I'm actually using normal peewee. This is still an issue for me, though. Maybe this will help you narrow it down.
I got same error in combination of asyncio
and peewee
:
Traceback (most recent call last):
File "AsyncTgClient.py", line 324, in StartClient
sim.MarkAsOffline()
File "../DbManager.py", line 110, in MarkAsOffline
self.save()
File "/usr/local/lib/python3.6/dist-packages/peewee.py", line 6399, in save
rows = self.update(**field_dict).where(self._pk_expr()).execute()
File "/usr/local/lib/python3.6/dist-packages/peewee.py", line 1821, in inner
return method(self, database, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/peewee.py", line 1892, in execute
return self._execute(database)
File "/usr/local/lib/python3.6/dist-packages/peewee.py", line 2369, in _execute
cursor = database.execute(self)
File "/usr/local/lib/python3.6/dist-packages/peewee.py", line 3034, in execute
return self.execute_sql(sql, params, commit=commit)
File "/usr/local/lib/python3.6/dist-packages/playhouse/shortcuts.py", line 215, in execute_sql
raise exc
File "/usr/local/lib/python3.6/dist-packages/playhouse/shortcuts.py", line 211, in execute_sql
return super(ReconnectMixin, self).execute_sql(sql, params, commit)
File "/usr/local/lib/python3.6/dist-packages/peewee.py", line 3028, in execute_sql
self.commit()
File "/usr/local/lib/python3.6/dist-packages/peewee.py", line 2796, in __exit__
reraise(new_type, new_type(*exc_args), traceback)
File "/usr/local/lib/python3.6/dist-packages/peewee.py", line 183, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.6/dist-packages/peewee.py", line 3024, in execute_sql
self.rollback()
File "/usr/local/lib/python3.6/dist-packages/peewee.py", line 3175, in rollback
return self._state.conn.rollback()
File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 429, in rollback
self._execute_command(COMMAND.COM_QUERY, "ROLLBACK")
File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 750, in _execute_command
raise err.InterfaceError("(0, '')")
peewee.InterfaceError: (0, '')
Same here. Peewee used in an asyncio method
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py", line 756, in _write_bytes
self._sock.sendall(data)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 3144, in execute_sql
cursor.execute(sql, params or ())
File "/usr/local/lib/python3.8/site-packages/pymysql/cursors.py", line 148, in execute
result = self._query(query)
File "/usr/local/lib/python3.8/site-packages/pymysql/cursors.py", line 310, in _query
conn.query(q)
File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py", line 547, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py", line 814, in _execute_command
self._write_bytes(packet)
File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py", line 759, in _write_bytes
raise err.OperationalError(
pymysql.err.OperationalError: (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/discord/client.py", line 343, in _run_event
await coro(*args, **kwargs)
File "/app/my_discord_client.py", line 96, in on_member_update
GameSessionManager.handle_user_update(before, after)
File "/app/game_session_manager.py", line 47, in handle_user_update
target_user = cls.get_or_create_user(user_id=after.id, name=after.name)
File "/app/game_session_manager.py", line 18, in get_or_create_user
target_user = DiscordUser.get(id=user_id)
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 6438, in get
return sq.get()
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 6884, in get
return clone.execute(database)[0]
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 1907, in inner
return method(self, database, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 1978, in execute
return self._execute(database)
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 2150, in _execute
cursor = database.execute(self)
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 3157, in execute
return self.execute_sql(sql, params, commit=commit)
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 3151, in execute_sql
self.commit()
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 2917, in __exit__
reraise(new_type, new_type(exc_value, *exc_args), traceback)
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 190, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 3144, in execute_sql
cursor.execute(sql, params or ())
File "/usr/local/lib/python3.8/site-packages/pymysql/cursors.py", line 148, in execute
result = self._query(query)
File "/usr/local/lib/python3.8/site-packages/pymysql/cursors.py", line 310, in _query
conn.query(q)
File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py", line 547, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py", line 814, in _execute_command
self._write_bytes(packet)
File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py", line 759, in _write_bytes
raise err.OperationalError(
peewee.OperationalError: (2006, "MySQL server has gone away (ConnectionResetError(104, 'Connection reset by peer'))")
2021-05-05 11:27:17,088 INFO [User update] name: Doucati, id: 426472859280343060
2021-05-05 11:27:17,088 INFO [Session skipped] User 'Doucati' stopped playing but was not tracked yet
2021-05-05 11:29:23,017 INFO [User update] name: justikiller, id: 237242555308179456
2021-05-05 11:29:23,017 INFO [Stopped playing] user: justikiller
Ignoring exception in on_member_update
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 3144, in execute_sql
cursor.execute(sql, params or ())
File "/usr/local/lib/python3.8/site-packages/pymysql/cursors.py", line 148, in execute
result = self._query(query)
File "/usr/local/lib/python3.8/site-packages/pymysql/cursors.py", line 310, in _query
conn.query(q)
File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py", line 547, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py", line 793, in _execute_command
raise err.InterfaceError(0, "")
pymysql.err.InterfaceError: (0, '')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/discord/client.py", line 343, in _run_event
await coro(*args, **kwargs)
File "/app/my_discord_client.py", line 96, in on_member_update
GameSessionManager.handle_user_update(before, after)
File "/app/game_session_manager.py", line 47, in handle_user_update
target_user = cls.get_or_create_user(user_id=after.id, name=after.name)
File "/app/game_session_manager.py", line 18, in get_or_create_user
target_user = DiscordUser.get(id=user_id)
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 6438, in get
return sq.get()
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 6884, in get
return clone.execute(database)[0]
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 1907, in inner
return method(self, database, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 1978, in execute
return self._execute(database)
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 2150, in _execute
cursor = database.execute(self)
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 3157, in execute
return self.execute_sql(sql, params, commit=commit)
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 3151, in execute_sql
self.commit()
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 2917, in __exit__
reraise(new_type, new_type(exc_value, *exc_args), traceback)
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 190, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.8/site-packages/peewee.py", line 3144, in execute_sql
cursor.execute(sql, params or ())
File "/usr/local/lib/python3.8/site-packages/pymysql/cursors.py", line 148, in execute
result = self._query(query)
File "/usr/local/lib/python3.8/site-packages/pymysql/cursors.py", line 310, in _query
conn.query(q)
File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py", line 547, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File "/usr/local/lib/python3.8/site-packages/pymysql/connections.py", line 793, in _execute_command
raise err.InterfaceError(0, "")
@Sispheor This call is not async, my guess is that sync fallback is used which is not intended:
target_user = DiscordUser.get(id=user_id)
Yes, sorry. I googled my error and fell here. It's an issue with Peewee.
face the same question , when I run my web application with peewee after 8 hours , it throw out" err.InterfaceError(0, "")" error
I am encountering this exception (using high-level api) then my asyncio based daemon is idle (about 15 minutes or so), after that time, this exception occurs, I have tried to close and reopen connection, and was surprised that Manager (high-level api) did not track its state (close/reopen is done using its methods).
after reopening a connection I've got stuck with a bunch of coroutines waiting for resolution which obviously will never come. =(
1) can it be a bug in pymysql/peewee 2) propose add connection status tracking to high-level Manager
any thoughts on how to fix that?