Document how to clear cache

khaeru commented 3 years ago

Upon upgrading some Python packages (I think including dogpile.cache), I began to see:

$ bugwarrior-pull --dry-run                                                                                                                                                   [1886/51844]
INFO:bugwarrior.db:Service-defined UDAs exist: you can optionally use the `bugwarrior-uda` command to export a list of UDAs you can add to your taskrc file.
INFO:bugwarrior.services:Starting to aggregate remote issues.
INFO:bugwarrior.services:Spawning 3 workers.
INFO:bugwarrior.services:Working on [github]
INFO:bugwarrior.services:Working on [github_reviews]
ERROR:bugwarrior.services:Worker for [github] failed: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
Traceback (most recent call last):
  File "/home/khaeru/.local/lib/python3.8/site-packages/bugwarrior/services/__init__.py", line 504, in _aggregate_issues
    for issue in service.issues():
  File "/home/khaeru/.local/lib/python3.8/site-packages/bugwarrior/services/github.py", line 479, in issues
    'annotations': self.annotations(tag, issue, issue_obj),
  File "/home/khaeru/.local/lib/python3.8/site-packages/bugwarrior/services/github.py", line 381, in annotations
    issue_obj.get_processed_url(url)
  File "/home/khaeru/.local/lib/python3.8/site-packages/bugwarrior/services/__init__.py", line 332, in get_processed_url
    return URLShortener().shorten(url)
  File "<decorator-gen-1>", line 2, in shorten
  File "/home/khaeru/.local/lib/python3.8/site-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func
    return self.get_or_create(
  File "/home/khaeru/.local/lib/python3.8/site-packages/dogpile/cache/region.py", line 1042, in get_or_create
    with Lock(
  File "/home/khaeru/.local/lib/python3.8/site-packages/dogpile/lock.py", line 185, in __enter__
    return self._enter()
  File "/home/khaeru/.local/lib/python3.8/site-packages/dogpile/lock.py", line 87, in _enter
    value = value_fn()
  File "/home/khaeru/.local/lib/python3.8/site-packages/dogpile/cache/region.py", line 977, in get_value
    value = self._get_from_backend(key)
  File "/home/khaeru/.local/lib/python3.8/site-packages/dogpile/cache/region.py", line 1264, in _get_from_backend
    return self._parse_serialized_from_backend(
  File "/home/khaeru/.local/lib/python3.8/site-packages/dogpile/cache/region.py", line 1221, in _parse_serialized_from_backend
    metadata = json.loads(bytes_metadata)
  File "/usr/lib/python3.8/json/__init__.py", line 343, in loads
    s = s.decode(detect_encoding(s), 'surrogatepass')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
INFO:bugwarrior.services:Done with [github] in 1.259314s

[…]

INFO:bugwarrior.services:Terminating workers
ERROR:bugwarrior.command:Aborted (critical error in target 'github')
Traceback (most recent call last):
  File "/home/khaeru/.local/lib/python3.8/site-packages/bugwarrior/command.py", line 73, in pull
    synchronize(issue_generator, config, main_section, dry_run)
  File "/home/khaeru/.local/lib/python3.8/site-packages/bugwarrior/db.py", line 357, in synchronize
    for issue in issue_generator:
  File "/home/khaeru/.local/lib/python3.8/site-packages/bugwarrior/services/__init__.py", line 572, in aggregate_issues
    raise RuntimeError(
RuntimeError: critical error in target 'github'

After poking around at the code for a while, I found and deleted the following files in ~/.cache:

dagd-py3.6.dbm
dagd-py3.7.dbm      
dagd-py3.8.dbm
dagd-py3.8.dbm.dogpile.lock
dagd-py3.8.dbm.rw.lock

The errors disappeared and bugwarrior-pull worked again.

The name of these files is non-obvious; I was looking for ~/.cache/bugwarrior or similar. It would be nice if the documentation included a description of this location, so that users could have some hope of finding and clearing the files to overcome such problems.

ryneeverett commented 3 years ago

I'd say this is more of a bug than a workaround we need to document. If it's commonly experienced and we can't figure out the underlying bug we could probably catch the exception and clear the cache through the python api.

ryneeverett commented 2 years ago

FYI, after 3b260104e2c513927b07a9a36724078932439d82 we should no longer be generating a new cache file for every minor version of python which should eliminate the collection seen here. That doesn't explain this error though.

GothenburgBitFactory / bugwarrior

Document how to clear cache #804