search

mborsetti / webchanges

webchanges anonymously checks web content (including images) and commands for changes, delivering instant notifications and AI-powered summaries to your favorite platform.
https://pypi.org/project/webchanges/
Other
37 stars 6 forks source link

storage.py KeyError #5

Closed Fedora7 closed 3 years ago

Fedora7 commented 3 years ago

webchanges 3.4.0 got the following error of after switching from urlwatch

Traceback (most recent call last): File "/home/user/.local/lib/python3.8/site-packages/webchanges/handler.py", line 90, in process self.load() File "/home/user/.local/lib/python3.8/site-packages/webchanges/handler.py", line 69, in load self.old_data, self.timestamp, self.tries, self.etag = self.cache_storage.load(guid) File "/home/user/.local/lib/python3.8/site-packages/webchanges/storage.py", line 659, in load return r['d'], timestamp, r['t'], r['e'] KeyError: 'd'

If I ran webchanges --clean-cache, it works. but running it one more time, it got the same error again.

I delete the .cache/webchanges/** and let it create the default database again, but it still run into this error

mborsetti commented 3 years ago

Thank you for the report and so sorry about the issue. It appears that for some reason on your system (testing is only done on Ubuntu and MacOS) the data is getting corrupted when saved, probably when copied from temporary to permanent storage at the end of the run, as the error is related to loading a saved snapshot.

Would you be so kind to repeat the testing with --verbose starting from an empty database (i.e. delete the .cache/webchanges/** and let it create the default database again) until you run into this error, and post the output? That may give some indications as to what may be going on.

mborsetti commented 3 years ago

@Fedora7 I have rewritten the code that copies the snapshots from temporary to permanent storage in SQL, which should be more robust, and published it as webchanges 3.4.1. The database is extensively tested in Ubuntu and MacOS, and passes all tests.

Hopefully that fixes your issue. If not, please do repeat the testing with --verbose starting from an empty database (i.e. delete the .cache/webchanges/** and let it create the default database again) until you run into this error, and post the output. That may give some indications as to what may be going on.

Fedora7 commented 3 years ago

I upgraded all the python packages including msgpack. The problem went away when running 3.4.1. I will do more testings laters. Thanks.

mborsetti commented 3 years ago

Sadly, I cannot replicate: everything works well both on Windows and Ubuntu. I enclose the Ubuntu logs. test3.txt

Please let me know if there are other differences other than my running in Ubuntu 20.10 instead of 20.04. The testing on GitHub Action is done in Ubuntu 20.04.2 LTS and it includes writing data to the database, closing it, reopening and reading it back, but ... corner cases!