search

release-engineering / dist-git

DistGit provides home for linux distribution packages.
Other
123 stars 38 forks source link

RFE: Replace lookaside cache with git-annex #1

Closed sorki closed 4 years ago

sorki commented 9 years ago

git-annex is designed for this task and would give us quite a flexibility compared to lookaside cache hack.

asamalik commented 9 years ago

I had the same idea. I was thinking about Git LFS: https://git-lfs.github.com/ I like it because it keeps the same work-flow for large files.

domcleal commented 9 years ago

If it's of interest, I added git-annex support to tito using locking/unlocking and we're using it in foreman-packaging.

LFS looks interesting from a usability standpoint, but I'm waiting for it to catch on a bit more, get packaged etc.

cgwalters commented 9 years ago

It'd be a lot more of a win to teach the stack about building from git repositories directly.

xsuchy commented 9 years ago

@cgwalters As co-maintainer of Tito and Mock - both tools which can build rpm directly from upstream git repository - I can tell you: this is impossible. To make this possible you will have to use some metadata in git repo (Tito does that) but people do not want to have it there ("because this is not just rpm project") or to have spec file on some preciously defined location ("but we do not want to have it there, we store it somewhere else").

cgwalters commented 9 years ago

https://wiki.gnome.org/Projects/GnomeContinuous accomplishes continuous delivery (building and shipping) from hundreds of upstream git repositories reliably and with a minimum of human intervention. It is very possible.

Specifically for metadata, look at: https://git.gnome.org/browse/gnome-continuous/log/manifest.json

It's where a few people can manage the build configuration of hundreds of upstream components. That's a major difference from the "one packager per package" model with lots of duplicate build rules, each owned individually.

Anyways it's not easy, and how some of the Continuous ideas would map to a package world is not trivial. But it is there, shipping 40-80+ times a day reliably directly from upstream git repos.

cgwalters commented 9 years ago

To be more concrete, for every project I maintain, I use the output of git archive (with recursive handling of submodules) as input for RPM's Source0. That's what I think the default should be. For those who have concerns over git's use of SHA1 etc, see http://permalink.gmane.org/gmane.comp.version-control.git/264533 which I hope to get back to soon.

sorki commented 9 years ago

@cgwalters

There's support for Source fetching in rpm (disabled by default), see https://github.com/rpm-software-management/rpm/blob/90976cf618cbeef1ab3dd557c0cfcff041330696/rpmio/rpmio.c#L433

I haven't got any further than initial research but it seems doable (especially with git evtag which looks very good).

Even if this is implemented we wouldn't be able to switch to building everything from git by default but it would get rid of git archiving overhead for projects that use git. There are still a lot of projects that release their packages as tarballs and this would require additional work to import these into git repositories.

I still think better integration of our build infra with git is the right way to go.

Ad Continuous - you can't really compare uniform project like Gnome with Fedora as maintaining build metadata for the former is much much easier. For another interesting approach take a look at NixOS and hydra build system which is able to rebuild packages automatically if any of the underlying deps change. They use single repository to store metadata for whole OS - here's an example of metadata for building a project directly from git https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/virtualization/euca2ools/default.nix (but even though they support building stuff directly from git majority of packages are still built from tarballs).

sorki commented 8 years ago

We now have tag signing in git: https://git-scm.com/book/tr/v2/Git-Tools-Signing-Your-Work

cgwalters commented 8 years ago

Tag signing has existed a long time. Commit signing is newer.

I maintain https://github.com/cgwalters/git-evtag which is intended to address any remaining conservatism about tag signing vs tarballs.

praiskup commented 4 years ago

I'm downvoting this for the same reasons I downvoted https://github.com/release-engineering/dist-git/issues/13 - dist-git is an "alternative" to git-annex. If we used git-annex e.g. on Fedora level, we could drop the dist-git entirely.

sorki commented 4 years ago

No problem though you should strive for interoperability with other tools and not force maintainers to git archive the sources. This is alleviated a bit by GitHub providing tarballs now but I bet you can still find tons of git archive comments in .spec files.