fs-repo-migrations is difficult to package

[Luflosi]

As can be seen at https://repology.org/project/fs-repo-migrations/versions, the latest fs-repo-migrations version in every repository is still 1.7.1. Version 2.0.1 was released on 01 Apr 2021, which is now close to one year old, yet no repository has updated to this version. Perhaps the reason for this is that it is very difficult to build fs-repo-migrations in a way that allows it to function without downloading binaries from the internet. The reasons for splitting fs-repo-migrations into many independent pieces mentioned at https://github.com/ipfs/fs-repo-migrations/issues/98 regarding growing size and poor maintainability seem valid but the way this was implemented and the seeming lack of documentation (I couldn't find what I needed) makes it difficult to build everything and make fs-repo-migrations use my own binaries instead of downloading them. Downloading binaries off the internet is not really in the spirit of software/Linux distributions, also see https://drewdevault.com/2021/09/27/Let-distros-do-their-job.html. I realize, that fs-repo-migrations is now "deprecated" according to https://github.com/ipfs/fs-repo-migrations/issues/98 but I don't see a good alternative. NixOS for example has code to optionally automatically run the migrations before starting the IPFS daemon: https://github.com/NixOS/nixpkgs/pull/136261/commits/8bce3034fda3b8d201961d382b55caaa54636e4d. This automatically upgrades the IPFS repo without requiring manual intervention. I don't see how else the same thing can be achieved. Please provide a way (and document that way) to build all the different pieces locally and allow fs-repo-migrations to use those pieces. If this is already possible, I apologize but I couldn't figure out how to do it. But I'm also absolutely not an expert in building Go packages. Thank you.

[Luflosi]

I think I figured it out. I built every migration individually and then added the resulting binaries to the PATH of fs-repo-migrations. See https://github.com/NixOS/nixpkgs/pull/161547 for how this looks in detail.

[lemmi]

The current layout really isn't very well supported with the go toolchain and also with the build helpers voidlinux uses. The more usual way to do things is to have a single module at the toplevel of the repository and mimic the usual $repo/cmd/fs-repo-$X-to-$Y layout. Keeping the current layout is proably also fine, but there needs to be a toplevel module, otherwise you manually need to change into every folder and do the build there instead of just being able to do a go build ./fs-repo-X-to-Y. Packing everything into a subdirectory should make it possible to run a single go build ./cmd/..., utilising available parallelism even better.

Additionally this desperately needs a new release.

https://github.com/void-linux/void-packages/pull/37775

[CMB]

As it stands now, the 11-to-12 migration won't run when built for Void Linux because of issue 156. Void only offers Go 1.19.

[ajakk]

Perhaps the reason for this is that it is very difficult to build fs-repo-migrations in a way that allows it to function without downloading binaries from the internet.

Note that the functionality of executing binaries downloaded from the internet introduces a security problem - if an attacker is able to gain control of the domain hosting the binaries relative to a target (e.g. through squatting on the domain or local DNS shenanigans), then that attacker can feed downloaders whatever binaries they want that then get executed by go-ipfs.

It seems like validating the checksums (at least) with checksums written into the application itself would mitigate this.