search

cpbotha / nvpy

Simplenote syncing note-taking application, inspired by Notational Velocity and ResophNotes, but uglier and cross-platformerer.
Other
849 stars 114 forks source link

RuntimeError: dictionary changed size during iteration #179

Closed lfini closed 3 years ago

lfini commented 5 years ago

After a few days of usage of the last github clone I get the following:

2019-04-07 16:03:11,194 - DEBUG - Starting full sync. 2019-04-07 16:03:11,229 - DEBUG - Retrieving full note list from server, could take a while. 2019-04-07 16:03:13,830 - DEBUG - Retrieved full note list from server. 2019-04-07 16:03:13,898 - ERROR - Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/nvpy-2.1.0.dev0-py3.6.egg/nvpy/notes_db.py", line 718, in wrapper sync_from_server_errors = self.sync_full_unthreaded() File "/usr/local/lib/python3.6/dist-packages/nvpy-2.1.0.dev0-py3.6.egg/nvpy/notes_db.py", line 795, in sync_full_unthreaded for lk in self.notes.keys(): RuntimeError: dictionary changed size during iteration

It looks like one thread is doing something to the dictionary ...

lfini commented 5 years ago

After a brief check I know it is a python3 issue. I'm gonna fixit

OliverLSanz commented 3 years ago

I'm also facing this error, it happens during every startup. The full log follows:

ERROR: An unexpected error occurred.
Thread(ident=139937152419648, name=MainThread)
Traceback (most recent call last):
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/nvpy.py", line 459, in observer_notes_db_error_sync_full
    raise evt.error
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/notes_db.py", line 780, in sync_full_unthreaded
    for ni, lk in enumerate(self.notes.keys()):
RuntimeError: dictionary changed size during iteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/debug.py", line 30, in wrapper
    return fn(*args, **kwargs)
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/tk.py", line 30, in wrapper
    return fn(*args, **kwargs)
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/nvpy.py", line 655, in observer_view_keep_house
    nsynced, sync_errors = self.notes_db.sync_to_server_threaded()
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/notes_db.py", line 722, in sync_to_server_threaded
    if float(o.note['syncdate']) > float(self.notes[okey]['syncdate']):
KeyError: '2456d72fb9e17a37430fc3a631b93f'

Other threads:
Thread(ident=139937152419648, name=MainThread)
  File "/home/oliver/.local/bin/nvpy", line 8, in <module>
    sys.exit(main())
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/nvpy.py", line 887, in main
    controller.main_loop()
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/nvpy.py", line 435, in main_loop
    self.view.main_loop()
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/view.py", line 1982, in main_loop
    self.root.mainloop()
  File "/usr/lib/python3.8/tkinter/__init__.py", line 1429, in mainloop
    self.tk.mainloop(n)
  File "/usr/lib/python3.8/tkinter/__init__.py", line 1892, in __call__
    return self.func(*args)
  File "/usr/lib/python3.8/tkinter/__init__.py", line 814, in callit
    func(*args)
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/view.py", line 2184, in fn
    callback()
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/nvpy.py", line 431, in poll_notifies
    self.notes_db.handle_notifies()
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/utils.py", line 222, in handle_notifies
    self.__invoke_observer(o, evt_type, evt)
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/utils.py", line 229, in __invoke_observer
    observer(self, event_type, event)
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/debug.py", line 30, in wrapper
    return fn(*args, **kwargs)
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/tk.py", line 30, in wrapper
    return fn(*args, **kwargs)
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/nvpy.py", line 472, in observer_notes_db_error_sync_full
    self.view.show_error('Sync error', emsg)
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/view.py", line 2106, in show_error
    tkMessageBox.showerror(title, msg)
  File "/usr/lib/python3.8/tkinter/messagebox.py", line 94, in showerror
    return _show(title, message, ERROR, OK, **options)
  File "/usr/lib/python3.8/tkinter/messagebox.py", line 72, in _show
    res = Message(**options).show()
  File "/usr/lib/python3.8/tkinter/commondialog.py", line 44, in show
    s = w.tk.call(self.command, *w._options(self.options))
  File "/usr/lib/python3.8/tkinter/__init__.py", line 1892, in __call__
    return self.func(*args)
  File "/usr/lib/python3.8/tkinter/__init__.py", line 814, in callit
    func(*args)
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/view.py", line 2184, in fn
    callback()
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/view.py", line 1630, in handler_housekeeper
    self.notify_observers('keep:house', None)
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/utils.py", line 205, in notify_observers
    self.__invoke_observer(o, evt_type, evt)
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/utils.py", line 229, in __invoke_observer
    observer(self, event_type, event)
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/debug.py", line 44, in wrapper
    all_threads=format_all_tracebacks(),
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/debug.py", line 17, in format_all_tracebacks
    stack = traceback.format_stack(frame)
Thread(ident=139937112553216, name=Thread-1)
  File "/usr/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/debug.py", line 30, in wrapper
    return fn(*args, **kwargs)
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/notes_db.py", line 997, in worker_save
    o = self.q_save.get()
  File "/usr/lib/python3.8/queue.py", line 170, in get
    self.not_empty.wait()
  File "/usr/lib/python3.8/threading.py", line 302, in wait
    waiter.acquire()
Thread(ident=139937104160512, name=Thread-2)
  File "/usr/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/debug.py", line 30, in wrapper
    return fn(*args, **kwargs)
  File "/home/oliver/.local/lib/python3.8/site-packages/nvpy/notes_db.py", line 1024, in worker_sync
    task = self.q_sync.get()
  File "/usr/lib/python3.8/queue.py", line 170, in get
    self.not_empty.wait()
  File "/usr/lib/python3.8/threading.py", line 302, in wait
    waiter.acquire()

> /home/oliver/.local/lib/python3.8/site-packages/nvpy/notes_db.py(722)sync_to_server_threaded()
-> if float(o.note['syncdate']) > float(self.notes[okey]['syncdate']):
(Pdb)
yuuki0xff commented 3 years ago

Thanks for reporting the bug.

Investigation about this bug postponed because it doesn't happen in my environment in a long time. I'll investigate and try to fix it.

yuuki0xff commented 3 years ago

Reproduce code:

# ~/.nvpy.cfg
[nvpy]
sn_username = test-account@example.com
sn_password = password
db_path = /var/tmp/nvpy-db
#!/bin/bash
# reproduce.sh
set -eux

rm -rf /var/tmp/nvpy-db
mkdir /var/tmp/nvpy-db
echo '
{
    "deleted": false,
    "modifydate": 20,
    "createdate": 10,
    "tags": [],
    "content": "modified note",
    "syncdate": 10
}
' |tee /var/tmp/nvpy-db/modified-note-{1..50}.json

python3 -m nvpy

# Press Ctrl+n during the full sync task in step 1.

Environment:

Patch:

diff --git a/nvpy/notes_db.py b/nvpy/notes_db.py
index 16b12c3..43e6529 100644
--- a/nvpy/notes_db.py
+++ b/nvpy/notes_db.py
@@ -811,7 +811,7 @@ def sync_full_unthreaded(self):
                         msg = "Sync step 1 error - Could not update note {0} to server: {1}".format(
                             key, str(result.error_object))
                         logging.error(msg)
-                        raise SyncError(msg)
+                        # raise SyncError(msg)

             # 2. Retrieves full note list from server.
             #    In phase 2 to 5, synchronized all notes from server to client.
yuuki0xff commented 3 years ago

I added mitigations for this issue in 844e8e5ad67cbae91603e95703b5fc34e78675ab. Reopen it if this issue recurs.