Caching behaviors

[ralt]

So, the fact that pkgfs implementations are a representation of the package manager at time T means several things:

• Either we build the whole tree at startup time and it gets outdated as soon as something changes. (what you do in pacman-fs)
• Or we return the relevant tree at call time, so it's always up-to-date, but is also very slow. (what I do at dpkg-fs)

Having a short-lived cache for some specific things (like some API calls) can be useful (e.g. I ls /pkg/installed/foo/files/, not having to call dpkg -L foo for every file is useful.), but it doesn't cover other, more important cases like ls /pkg/installed.

I think I can get a cache based on the mtime of /var/lib/dpkg/lock, can you also get something similar with libalpm?

[ralt]

On debian, there's actually 2 locks: one for apt (the index), another for dpkg. Which means I can have 2 caches: one for /pkg/index, and another for /pkg/installed, based on the mtimes of both /var/lib/apt/lists/lock and /var/lib/dpkg/lock.

I need to investigate more though, as this is the theory. The /var/lib/apt/lists/lock file mtime is from July, as I'm speaking on my machine. So I need to confirm this. (I just confirmed that the /var/lib/dpkg/lock's mtime changes when new packages are installed/removed.)

But anyway, this sounds like a good option for me altogether.

The question is: can you get something similar?

[ralt]

Aaaand I found the file updated on every apt-get update (i.e. the one to base /pkg/index on): /var/cache/apt/pkgcache.bin

[Zirak]

This is a difficult problem.

The naive solution is to cache optimistically and build the tree upfront and only invalidate when a destructive action is taken by the user, like running /pkg/sync. This has the enormous downside of failing to catch destructive actions not done via pkgfs but from the package manager proper, as in a user running apt-get update.

Another suggestion is, as you said, to monitor files whose changes are indicative to our dbs changing, and invalidate the cache on such actions. This can be done via inotify or stat polling or even lazily, when the user triggers an action.

I don't particularly like the second suggestion, but it's loads better than the first. Later on I'll check if it's feasible to implement under pacman, but again, other package managers should be taken into consideration. (aside: should they? this is an implementation issue, and while it's good to discuss, each implementation should tackle it as it wants; the spec merely needs to specify that the filesystem needs to stay current and up-to-date)

[ralt]

I'm not suggesting to watch the file for change, but rather to use a caching method that will check the mtime of these files every time it's called.