search

vodik / repose

an archlinux repo poking tool
GNU General Public License v2.0
128 stars 15 forks source link

Delta support #43

Open barthalion opened 7 years ago

barthalion commented 7 years ago

It would be great if repose could also generate or at least add delta files to the database.

vodik commented 7 years ago

Yeah, should probably get around to getting that. I'll see what I can do...

vodik commented 7 years ago

Started hacking around with this on the delta-support branch.

vodik commented 7 years ago

Okay, so as a start, its fairly straightforward to add support to serialise delta information. This, at the very least, means that I don't lose that information.

But I never really stopped to think about how to support deltas in this tool. I guess I need to, when calculating which files trigger an update, generate a delta between the old version an the new version, and record that information.

Few open questions/musing out loud):

  1. Should I automatically remove old deltas? Only one? More? Calculate when the deltas exceed the package's size? I don't actually know how well pacman supports deltas tbh.
  2. Should I sign deltas I produce?
  3. Where to the deltas go, when the package pool and the repository root don't match. Should they be symlinked?
  4. Are there cases where I shouldn't generate deltas? Like in the case when I detect I need to rebuild the database because package checksum/timestamp mismatch (but no version bump), I guess making a delta between those releases don't make sense, and all existing deltas have to be invalidated.

And random footnote, looks like pacman expects deltas to have a md5 checksum. I wonder if there's support for anything else... or if that should also get added upstream.

vodik commented 7 years ago

Well, delta serialisation support is on master, there's still a long road ahead though...

barthalion commented 7 years ago

Should I automatically remove old deltas? Only one? More? Calculate when the deltas exceed the package's size? I don't actually know how well pacman supports deltas tbh.

I don't know either to be honest. Depends if it can chain applying deltas from locally cached version that is no longer available in the repository to the latest version. If it can, I guess no delta should be removed automatically.

Should I sign deltas I produce?

No idea if it supports signed deltas…

Where to the deltas go, when the package pool and the repository root don't match. Should they be symlinked?

I didn't notice that the package pool can be in different directory now. I guess it should be in the pool?

On a side note, I didn't expect you to implement it overnight; it's just something that would be nice to have. From what I see know, delta support in pacman is terribly slow for the biggest packages (and for small ones it makes no sense to use it). I guess it differs from the yum/dnf implementation where it works quite okay.

vodik commented 7 years ago

Yeah, its not coming overnight, that's for sure. I'll need to play with it.

From what I see know, delta support in pacman is terribly slow for the biggest packages

I've avoided it actually because my understanding is its basically not really supported. I don't think anyone wants to use deltas on the official repos and thus noone is really auditing or testing that code.

I mean, with all the discussion about abandoning md5sum by default (which repose doesn't offer anyways), deltas are still using a md5 checksum. Granted, I think the resulting file from the applied delta is still compared to the appropriate signatures, so its not that bad.