[Packaging] System package manager-friendly vendoring

[leycec]

Greetings, glorious mangal maintainers! I quietly package awesome software for Gentoo Linux. Because mangal is clearly both awesome and software, packaging mangal for Gentoo is what I am now doing.

Except... I can't. Cue sadness cat.

mangal: We Don't Need No "vendor/".

The mangal repository (and thus source tarballs derived from this repository) currently omits the top-level vendor/ directory required to reproducibly build and install mangal from source.

That's usually fine. In the absence of a vendor/ directory, go subcommands silently download and cache dependencies listed in go.mod into the user-local module cache (e.g., ~/go/pkg/mod/). Go developers and brave end users manually building mangal from scratch are accustomed to that.

That's less fine for foolhardy packagers (such as myself) attempting to package mangal for source-based redistribution at the system level. But "source-based redistribution at the system level" is the literal definition of Gentoo Linux. In other words, Gentoo isn't happy. Let's make Gentoo happy.

Gentoo: Actually, We Really Need "vendor/".

Most system-level package managers (e.g., emerge under Gentoo Linux) and language-level package managers (e.g., the Swift Package Manager (SPM)) sandbox at least filesystem access for security. Increasingly many also sandbox network access for the same reason. This is where our packaging trouble starts and ends.

Since mangal omits a vendor/ directory, go subcommands silently:

• Download dependencies, thus violating network sandboxing during installation.
• Cache dependencies, thus violating filesystem sandboxing during installation.

Admittedly, Gentoo can trivially circumvent filesystem sandboxing by just (A) caching to a temporary non-sandboxed work directory while building and then (A) deleting that directory after building. That's fine.

Gentoo cannot trivially circumvent network sandboxing, however. In theory, Gentoo could vendor mangal dependencies into a new source tarball that we host and maintain on your behalf... which is the problem. Gentoo ^{...okay, I mean me} doesn't want to host and maintain separate source vendor tarballs for every single Go application in existence.

We're lazy and slovenly. Vendoring Go dependencies is non-trivial, error-prone, time-consuming, and anti-fun. Moreover, every single other Linux distribution (of which there are a countably infinite number) would need to reduplicate that same work. This is probably why most Linux distributions that currently package mangal sidestep this issue by just redistributing your binary Linux executables. That is what Arch does, for example. But that's bad – or at least fundamentally incompatible with the Gentoo religion.

We're zealous fanatics about source-based installation. We couldn't be wrong! :sweat_smile:

mangal: Yah. We're Still Unconvinced.

Omitting vendor/ is typically the best practice for Go middleware like libraries and frameworks. Conversely, committing vendor/ is typically the best practice for actual Go applications shipping executable binaries... like mangal. Sources or it didn't happen, so:

• Should You Commit the Vendor Folder in Go? Conclusion:

The answer to the age-old question of, “should I commit the vendor dependencies in my Go project to source control?” is “almost always.“

• To vendor, or not to vendor. Conclusion:

For my own peace of mind I would always prefer to commit the vendor/ directory alongside go.mod and go.sum...

mangal: We Still Don't Care.

Very well. I can see that neither the lolcat nor the two blog articles I linked to above have convinced you. In that case, I shall now bludgeon you with verbose words.

There's really only one valid reasons to omit vendor/: repository size. But that's not a particularly valid reason in the modern era, because GitHub ^{...so, Microsoft is what I'm saying} is already committed to taking the bandwidth and storage hit on your behalf.

There are many valid reasons to commit vendor/, however – including:

• Compatibility with package managers that sandbox. Gentoo. Always Gentoo.
• Deterministic builds. mangal builds are currently non-deterministic. By deferring to go subcommands that bang on remote servers for online installation, mangal builds tacitly assume that URLs, versions, and licenses are perfectly stable across all dependencies – forever. But that's never the case. Packaging is a constantly shifting landscape wrapped in a badly documented enigma inside a vanishing mystery of 404 errors. The Node.js world discovered this the rough way when left-pad spontaneously blew everything up.
• Better faster harder CI/D. Continuous integration (CI) and deployment (CD) workflows suddenly become substantially faster and more robust, because go no longer needs to bang on remote servers for online installation. Since requisite dependencies are locally available under vendor/, CI/D turnaround (and thus turnaround for tests and PRs) measurably improves.

Gentoo: I Am Very Tired Now.

Thanks so much for all the automated manga downloading, mangal magicians. My manga backlog may currently be crying, but everyone here is a spectacular gift to humanity.

[metafates]

Wooah! Firstly, thank you for all the kind words, really appreciate it! Secondly, your points are very reasonable, you convinced me by 110% 😁 From now on I'll vendor for all my go projects

Just to be sure that I won't do anything wrong - all I have to do to is to run this?

go mod vendor

Or there's something extra needs to be done?

[leycec]

Aww! You so nice, @metafates. And might I add that that's the most frightening GitHub avatar I've seen today. I commend you.

Just to be sure that I won't do anything wrong - all I have to do to is to run this?

I'm... pretty sure. But I'm uncertain. Python is my backyard. Go is merely where I play on the weekends. If mangal depends on anything that compiles C- or C++-based dependencies, even go mod vendor might not suffice due to an unresolved issue in Go itself. Let's pretend that issue doesn't exist.

Thanks for being so accommodating, too. What a stunningly beautiful CLI app you've made here. :open_mouth:

[metafates]

Yep, mangal does not depend on any C code, so I guess that's it. I'll roll vendored version soon. Thank you so much =)

[metafates]

vendor/ is in the main, should be ok now!

[leycec]

Wow. So much amazing enthusiasm! You make binging manga even more fun, @metafates.

As soon as I package mangal for Gentoo with a working ebuild at my overlay, I'll submit a PR here adding Gentoo-specific installation instructions to your README.md. Until then, let's read manga until we pass out at 5:32 AM on a Wednesday evening again.

[metafates]

Haha, okay, I'll be waiting 😁