py2 LazyDB/FilesDB: Use pickle protocol v2, write gdbm-backed shelve, set FILESDB_FORMAT_VERSION = 3

[joebonrichie]

The goal is to prototype auto-regeneration of a gdbm backed shelve in eopkg.py2.

Once we're happy with the logic, we can implement similar logic for eopkg.py3 (see #89), thus enabling us to switch back and forth between versions and having the FilesDB cache auto-regenerated on demand.

eopkg.py2 sets pisi/db/filesdb.FILESDB_FORMAT_VERSION = 3 manually.

Test Plan

Non-root ./eopkg.py2 behaviour:

1. No FilesDB: sudo rm /var/lib/eopkg/info/files.db* ; time ./eopkg.py2 -d sf /bin/bash
2. Empty FilesDB: sudo touch /var/lib/eopkg/info/files.db && time ./eopkg.py2 -d sf /bin/bash
3. Corrupt FilesDB: sync && echo "0xDEADBEEF" | sudo tee /var/lib/eopkg/info/files.db && sync && time ./eopkg.py2 -d sf /bin/bash
4. Unversioned bsddb FilesDB: sudo rm /var/lib/eopkg/info/files.db* ; sync && time sudo eopkg.py2 -dy rdb && sync && time ./eopkg.py2 -d sf /bin/bash

Root ./eopkg.py2 behaviour:

1. Rebuild FilesDB: sync && time sudo ./eopkg.py2 -dy rdb
2. No FilesDB: sync && sudo rm /var/lib/eopkg/info/files.db* ; time sudo ./eopkg.py2 -d sf /bin/bash && time sudo ./eopkg.py2 -d sf /bin/bash
3. Empty FilesDB: sync && sudo rm /var/lib/eopkg/info/files.db* ; sudo touch /var/lib/eopkg/info/files.db && time sudo ./eopkg.py2 -d sf /bin/bash && sync && time sudo ./eopkg.py2 -d sf /bin/bash
4. Corrupt FilesDB: sync && echo "0xDEADBEEF" | sudo tee /var/lib/eopkg/info/files.db && sync && time sudo ./eopkg.py2 -d sf /bin/bash && sync && time sudo ./eopkg.py2 -d sf /bin/bash
5. Unversionved bsddb FilesDB: sync && sudo rm /var/lib/eopkg/info/files.db* ; sync && time sudo eopkg.py2 -dy rdb && sync && time sudo ./eopkg.py2 -d sf /bin/bash && sync && time sudo ./eopkg.py2 -d sf /bin/bash

NB: All instances of sudo eopkg.py2 -dy rdb are quite deliberate; we want the old version of eopkg.py2 to generate a bsddb format FilesDB db, so we can see what happens and whether it gets regenerated as it is supposed to when we run the present PR code against it.

Land before #89

[sheepman4267]

So, I'm not sure what's going on here. The documented test plan here seems to work, but unfortunately running sudo ./eopkg.py2 rdb does not. It looks like it finishes rebuilding the DB (maybe), but then dies like this at the end:

Adding 'sqlite3-devel' to db... DEBUG: InstallDB initialized in 0.00866103172302.
No handlers could be found for logger "pisi"
System error. Program terminated.
<type 'exceptions.AttributeError'>: installed_db
Please use 'eopkg help' for general help.

Traceback:
  File "./eopkg.py2", line 78, in <module>
    cli.run_command()
  File "/home/sheepman/Solus/eopkg/pisi/cli/pisicli.py", line 138, in run_command
    self.command.run()
  File "/home/sheepman/Solus/eopkg/pisi/cli/rebuilddb.py", line 50, in run
    pisi.api.rebuild_db(ctx.get_option('files'))
  File "/home/sheepman/Solus/eopkg/pisi/api.py", line 65, in wrapper
    ret = func(*__args,**__kw)
  File "/home/sheepman/Solus/eopkg/pisi/api.py", line 855, in rebuild_db
    rebuild_filesdb()
  File "/home/sheepman/Solus/eopkg/pisi/api.py", line 834, in rebuild_filesdb
    files = installdb.get_files(pkg)
  File "/home/sheepman/Solus/eopkg/pisi/db/installdb.py", line 161, in get_files
    files_xml = os.path.join(self.package_path(package), ctx.const.files_xml)
  File "/home/sheepman/Solus/eopkg/pisi/db/installdb.py", line 355, in package_path
    if self.installed_db.has_key(package):
  File "/home/sheepman/Solus/eopkg/pisi/db/lazydb.py", line 127, in __getattr__
    raise AttributeError, attr

I tried for about half an hour to root-cause this, but don't have a lot of brain-power tonight. The fault doesn't immediately seem to come from changes made in the PR, and I think PiSi doesn't quite print debug messages in order. Makes instrumenting this really, really annoying...

[silkeh]

@sheepman4267 does that also happen on the python2 branch?

[ermo]

@sheepman4267 First of all, thank you for being diligent enough to think of testing the rdb command.

Looking at the trace with my knowledge of the code, the order of operations looks legit.

This clearly needs fixing. I have an idea for how to refactor this so it's less spaghetti, and so the order of operations are saner.

It should essentially make rdb nicer in terms of its call chain and move the responsibility for the why, the when and the how of the rebuild fully into the FilesDB class.

[ermo]

@sheepman4267 does that also happen on the python2 branch?

@silkeh It doesn't for me.