joelpet commented 3 years ago

Issue report

Operating system: Arch Linux

App version: 2020.7 (currently installed) App version: 2021.1 (trying to build)

Issue description

I am unable to build the package for Arch Linux with yay using https://aur.archlinux.org/packages/mullvad-vpn/.

There seems to be an issue with Go module versioning of the golang.zx2c4.com/wireguard dependency.

❯ yay
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 multilib is up to date
:: Starting full system upgrade...
 there is nothing to do
:: Searching databases for updates...
:: Searching AUR for updates...
 -> Orphaned AUR Packages:  empathy
 -> Flagged Out Of Date AUR Packages:  debhelper
:: 1 Packages to upgrade.
1  aur/mullvad-vpn  2020.7-1 -> 2021.1-1
==> Packages to exclude: (eg: "1 2 3", "1-3", "^4" or repo name)
==> 
:: Checking for conflicts...
:: Checking for inner conflicts...
[Aur:1]  mullvad-vpn-2021.1-1

  1 mullvad-vpn                      (Installed) (Build Files Exist)
==> Packages to cleanBuild?
==> [N]one [A]ll [Ab]ort [I]nstalled [No]tInstalled or (1 2 3, 1-3, ^4)
==> a
:: Deleting (1/1): /tmp/yaytmp-1002/mullvad-vpn
:: Downloaded PKGBUILD (1/1): mullvad-vpn
  1 mullvad-vpn                      (Installed) (Build Files Exist)
==> Diffs to show?
==> [N]one [A]ll [Ab]ort [I]nstalled [No]tInstalled or (1 2 3, 1-3, ^4)
==> 
:: (1/1) Parsing SRCINFO: mullvad-vpn
==> Making package: mullvad-vpn 2021.1-1 (sön 14 feb 2021 11:50:07)
==> Retrieving sources...
  -> Cloning mullvadvpn-app git repo...
Cloning into bare repository '/tmp/yaytmp-1002/mullvad-vpn/mullvadvpn-app'...
remote: Enumerating objects: 2737, done.
remote: Counting objects: 100% (2737/2737), done.
remote: Compressing objects: 100% (1406/1406), done.
remote: Total 148528 (delta 1766), reused 1331 (delta 1331), pack-reused 145791
Receiving objects: 100% (148528/148528), 40.98 MiB | 7.15 MiB/s, done.
Resolving deltas: 100% (94075/94075), done.
  -> Cloning mullvadvpn-app-binaries git repo...
Cloning into bare repository '/tmp/yaytmp-1002/mullvad-vpn/mullvadvpn-app-binaries'...
remote: Enumerating objects: 213, done.
remote: Counting objects: 100% (213/213), done.
remote: Compressing objects: 100% (149/149), done.
remote: Total 1706 (delta 86), reused 125 (delta 57), pack-reused 1493
Receiving objects: 100% (1706/1706), 184.64 MiB | 6.14 MiB/s, done.
Resolving deltas: 100% (1007/1007), done.
  -> Found mullvad-vpn.sh
==> Validating source files with sha256sums...
    mullvadvpn-app ... Skipped
    mullvadvpn-app-binaries ... Skipped
    mullvad-vpn.sh ... Passed
==> Verifying source file signatures with gpg...
    mullvadvpn-app git repo ... Passed
    mullvadvpn-app-binaries git repo ... Passed
==> Making package: mullvad-vpn 2021.1-1 (sön 14 feb 2021 11:50:53)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
  -> Updating mullvadvpn-app git repo...
Fetching origin
  -> Updating mullvadvpn-app-binaries git repo...
Fetching origin
  -> Found mullvad-vpn.sh
==> Validating source files with sha256sums...
    mullvadvpn-app ... Skipped
    mullvadvpn-app-binaries ... Skipped
    mullvad-vpn.sh ... Passed
==> Verifying source file signatures with gpg...
    mullvadvpn-app git repo ... Passed
    mullvadvpn-app-binaries git repo ... Passed
==> Removing existing $srcdir/ directory...
==> Extracting sources...
  -> Creating working copy of mullvadvpn-app git repo...
Cloning into 'mullvadvpn-app'...
done.
Switched to a new branch 'makepkg'
  -> Creating working copy of mullvadvpn-app-binaries git repo...
Cloning into 'mullvadvpn-app-binaries'...
done.
Switched to a new branch 'makepkg'
==> Starting prepare()...
Submodule 'dist-assets/binaries' (https://github.com/mullvad/mullvadvpn-app-binaries) registered for path 'dist-assets/binaries'
Cloning into '/tmp/yaytmp-1002/mullvad-vpn/src/mullvadvpn-app/dist-assets/binaries'...
Submodule path 'dist-assets/binaries': checked out 'fa76f058d6f5fa66e62f9c4a291e6079cea22e37'
Removing old Rust build artifacts
==> Sources are ready.
==> Making package: mullvad-vpn 2021.1-1 (sön 14 feb 2021 11:51:23)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> WARNING: Using existing $srcdir/ tree
==> Starting build()...
Building Mullvad VPN 2021.1...
Updating version in metadata files...
v2021.1.0
Building wireguard-go...
/tmp/yaytmp-1002/mullvad-vpn/src/mullvadvpn-app/wireguard/libwg /tmp/yaytmp-1002/mullvad-vpn/src/mullvadvpn-app
get "golang.zx2c4.com/wireguard": found meta tag get.metaImport{Prefix:"golang.zx2c4.com/wireguard", VCS:"git", RepoRoot:"https://git.zx2c4.com/wireguard-go"} at //golang.zx2c4.com/wireguard?go-get=1
go: golang.zx2c4.com/wireguard@v0.0.20201118: reading golang.zx2c4.com/wireguard/go.mod at revision v0.0.20201118: unknown revision v0.0.20201118
==> ERROR: A failure occurred in build().
    Aborting...
error making: mullvad-vpn

I then tried the following:

Update go.mod with the tag name available in the Git repository:
```
# go.mod:
...
golang.zx2c4.com/wireguard 0.0.20201118
...
```
Run go get to have Go resolve the proper Go modules version for that tag. Resolving the Git tag succeeds, but the command fails because wireguard-windows@v0.3.1 also references the unknown revision (according to my Go toolchain) v0.0.20201118.
```
❯ go get
go: golang.zx2c4.com/wireguard/windows@v0.3.1 requires
golang.zx2c4.com/wireguard@v0.0.20201118: reading golang.zx2c4.com/wireguard/go.mod at revision v0.0.20201118: unknown revision v0.0.20201118
```

go.mod:

... golang.zx2c4.com/wireguard v0.0.0-20201118132417-da19db415a58 ...


My Go version:

❯ go version go version go1.15.8 linux/amd64

pinkisemils commented 3 years ago

You could try using our own buildscript for building wireguard-go. Since that one most certainly works.

pinkisemils commented 3 years ago

Yep, just using our script seems to work just fine. You can have a look at my branch for the diff.

joelpet commented 3 years ago

I run into the same problem with https://github.com/pinkisemils/mullvad-vpn:

mullvad-vpn on  try-using-mullvad-wggo-script 
❯ makepkg
==> Making package: mullvad-vpn 2021.1-1 (mån 15 feb 2021 19:48:29)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
  -> Cloning mullvadvpn-app git repo...
Cloning into bare repository '/home/joelpet/src/github.com/pinkisemils/mullvad-vpn/mullvadvpn-app'...
remote: Enumerating objects: 3176, done.
remote: Counting objects: 100% (3176/3176), done.
remote: Compressing objects: 100% (1647/1647), done.
remote: Total 148684 (delta 1966), reused 1652 (delta 1484), pack-reused 145508
Receiving objects: 100% (148684/148684), 41.39 MiB | 13.73 MiB/s, done.
Resolving deltas: 100% (94087/94087), done.
  -> Cloning mullvadvpn-app-binaries git repo...
Cloning into bare repository '/home/joelpet/src/github.com/pinkisemils/mullvad-vpn/mullvadvpn-app-binaries'...
remote: Enumerating objects: 213, done.
remote: Counting objects: 100% (213/213), done.
remote: Compressing objects: 100% (149/149), done.
remote: Total 1706 (delta 86), reused 125 (delta 57), pack-reused 1493
Receiving objects: 100% (1706/1706), 184.64 MiB | 13.68 MiB/s, done.
Resolving deltas: 100% (1007/1007), done.
  -> Found mullvad-vpn.sh
==> Validating source files with sha256sums...
    mullvadvpn-app ... Skipped
    mullvadvpn-app-binaries ... Skipped
    mullvad-vpn.sh ... Passed
==> Verifying source file signatures with gpg...
    mullvadvpn-app git repo ... Passed
    mullvadvpn-app-binaries git repo ... Passed
==> Extracting sources...
  -> Creating working copy of mullvadvpn-app git repo...
Cloning into 'mullvadvpn-app'...
done.
Switched to a new branch 'makepkg'
  -> Creating working copy of mullvadvpn-app-binaries git repo...
Cloning into 'mullvadvpn-app-binaries'...
done.
Switched to a new branch 'makepkg'
==> Starting prepare()...
Submodule 'dist-assets/binaries' (https://github.com/mullvad/mullvadvpn-app-binaries) registered for path 'dist-assets/binaries'
Cloning into '/home/joelpet/src/github.com/pinkisemils/mullvad-vpn/src/mullvadvpn-app/dist-assets/binaries'...
Submodule path 'dist-assets/binaries': checked out 'fa76f058d6f5fa66e62f9c4a291e6079cea22e37'
Removing old Rust build artifacts
==> Starting build()...
Building Mullvad VPN 2021.1...
Updating version in metadata files...
v2021.1.0
Building wireguard-go
Building wireguard-go for Linux
~/src/github.com/pinkisemils/mullvad-vpn/src/mullvadvpn-app/wireguard/libwg ~/src/github.com/pinkisemils/mullvad-vpn/src/mullvadvpn-app/wireguard
get "golang.zx2c4.com/wireguard": found meta tag get.metaImport{Prefix:"golang.zx2c4.com/wireguard", VCS:"git", RepoRoot:"https://git.zx2c4.com/wireguard-go"} at //golang.zx2c4.com/wireguard?go-get=1
get "golang.zx2c4.com/wireguard/windows": found meta tag get.metaImport{Prefix:"golang.zx2c4.com/wireguard/windows", VCS:"git", RepoRoot:"https://git.zx2c4.com/wireguard-windows"} at //golang.zx2c4.com/wireguard/windows?go-get=1
get "golang.org/x/sys": found meta tag get.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at //golang.org/x/sys?go-get=1
get "golang.org/x/text": found meta tag get.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at //golang.org/x/text?go-get=1
get "golang.org/x/net": found meta tag get.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at //golang.org/x/net?go-get=1
get "golang.org/x/crypto": found meta tag get.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at //golang.org/x/crypto?go-get=1
get "golang.org/x/term": found meta tag get.metaImport{Prefix:"golang.org/x/term", VCS:"git", RepoRoot:"https://go.googlesource.com/term"} at //golang.org/x/term?go-get=1
get "golang.org/x/tools": found meta tag get.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at //golang.org/x/tools?go-get=1
go: golang.zx2c4.com/wireguard@v0.0.20201118: reading golang.zx2c4.com/wireguard/go.mod at revision v0.0.20201118: unknown revision v0.0.20201118
==> ERROR: A failure occurred in build().
    Aborting...

Presumably, it's related to the specific configuration of the Go build tools in my environment; I was able to successfully build the 2021.1-1 AUR version on another computer running Arch Linux with similar configuration.

However, I'm still puzzled by the versioning of the golang.zx2c4.com/wireguard module; according to the Go modules docs, "[e]ach version starts with the letter v, followed by a semantic version." But in the wireguard case, the only tags available are plain semantic versions without the 'v' prefix.

Furthermore, "Go may access modules in version control systems using tags, branches, or revisions that don't follow these conventions. However, within the main module, the go command will automatically convert revision names that don't follow this standard into canonical versions." I believe this is what I observed when go.mod was updated with golang.zx2c4.com/wireguard v0.0.0-20201118132417-da19db415a58 when running go get after having manually set the version to 0.0.20201118. This, I believe, is an indication that the naked semantic version Git tags are indeed not recognized as Go module versions.

With all that said, I realize that this is an upstream issue, if anything. Thank you for your time and effort looking into this.

joelpet commented 3 years ago

For the record, as a workaround for now, I built the package inside a Docker container:

FROM archlinux:base-devel

RUN install -Dm440 <(echo "%wheel ALL=(ALL) NOPASSWD: ALL") "/etc/sudoers.d/49-nopasswd"

RUN pacman -Sy

RUN useradd --create-home --groups wheel builder

USER builder

WORKDIR /home/builder

RUN curl --silent --show-error --output - https://aur.archlinux.org/cgit/aur.git/snapshot/mullvad-vpn.tar.gz | tar -xzf-

RUN gpg --recv-keys EA0A77BF9E115615FC3BD8BC7653B940E494FE87 8339C7D2942EB854E3F27CE5AEE9DECFD582E984

WORKDIR mullvad-vpn

RUN makepkg --syncdeps --rmdeps --noconfirm

USER root

RUN mkdir /tmp/mullvad-vpn && cp mullvad-vpn-*.pkg.tar.zst /tmp/mullvad-vpn/

And then extracted and installed the package:

sudo docker run --rm -v $PWD:/mnt mullvadvpnarchlinux:latest bash -c 'cp /tmp/mullvad-vpn/* /mnt'
sudo pacman -U mullvad-vpn-2021.1-1-x86_64.pkg.tar.zst

pinkisemils commented 3 years ago

Firstly, I apologize for the premature closure of the issue - clearly nothing's resolved. That's really weird, what Go version were you using? The dependency management has churned for quite some while now, but as far as I was aware, it settled down to the point where a plain go build should just work. Bear in mind, we only seem to test Go version 1.13.6. If you're using something significantly newer or older, that might be the root of the issues. If it's a version that's significantly newer, I'd like to know - that's something we'll have to fix.

As far as wireguard-go's versioning goes, that's just what the author of wireguard-go does, and it's worked so far.

joelpet commented 3 years ago

what Go version were you using?
go version go1.15.8 linux/amd64

I find it interesting that the build inside the Docker container works fine, just like building it on my other Arch Linux x86_64 machine, but with my current user on this machine it doesn't. I also tried creating a new user on this machine, and with that user it works. That makes me think that there is simply some configuration difference causing the issue. I just haven't had time yet to figure out exactly what that would be.

For the record, v2021.2.0 fails in the same way:

==> Starting build()...
Building Mullvad VPN 2021.2...
Updating version in metadata files...
v2021.2.0
Building wireguard-go...
/tmp/yaytmp-1002/mullvad-vpn/src/mullvadvpn-app/wireguard/libwg /tmp/yaytmp-1002/mullvad-vpn/src/mullvadvpn-app
get "golang.org/x/sys": found meta tag get.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at //golang.org/x/sys?go-get=1
get "golang.zx2c4.com/wireguard/windows": found meta tag get.metaImport{Prefix:"golang.zx2c4.com/wireguard/windows", VCS:"git", RepoRoot:"https://git.zx2c4.com/wireguard-windows"} at //golang.zx2c4.com/wireguard/windows?go-get=1
get "golang.zx2c4.com/wireguard": found meta tag get.metaImport{Prefix:"golang.zx2c4.com/wireguard", VCS:"git", RepoRoot:"https://git.zx2c4.com/wireguard-go"} at //golang.zx2c4.com/wireguard?go-get=1
get "golang.org/x/crypto": found meta tag get.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at //golang.org/x/crypto?go-get=1
get "golang.org/x/text": found meta tag get.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at //golang.org/x/text?go-get=1
get "golang.org/x/net": found meta tag get.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at //golang.org/x/net?go-get=1
get "golang.org/x/term": found meta tag get.metaImport{Prefix:"golang.org/x/term", VCS:"git", RepoRoot:"https://go.googlesource.com/term"} at //golang.org/x/term?go-get=1
get "golang.org/x/tools": found meta tag get.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at //golang.org/x/tools?go-get=1
go: golang.zx2c4.com/wireguard@v0.0.20201118: reading golang.zx2c4.com/wireguard/go.mod at revision v0.0.20201118: unknown revision v0.0.20201118
==> ERROR: A failure occurred in build().
    Aborting...
error making: mullvad-vpn

yochananmarqos commented 3 years ago

Do you have any Go environment variables setup? If so, that would explain why it works in a Docker container but not with Yay. Build it in a clean chroot.

joelpet commented 3 years ago

Do you have any Go environment variables setup?
$ printenv | grep -i go
PATH=[redacted]:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
Build it in a clean chroot.

To diagnose the issue? Or as a permanent solution?

I am fairly sure that there is something in my specific user's environment that is causing the issue, based on that the build inside the Docker container using an Arch Linux image works, plus it works for a newly set up user on the same machine.

yochananmarqos commented 3 years ago

Building in a chroot will allow you to build without worrying about your environment variables.

joelpet commented 3 years ago

Thank you for the advice. Building in a chroot indeed makes the build succeed – perhaps unsurprisingly, in this case. I realize this would be the most reliable way of building packages from the AUR, but the convenience of simply running yay to upgrade packages is very compelling. I will try to narrow down the reason for the observed behavior of Go – out of curiosity, if nothing else – and report back if I find something.

Feel free to close this ticket, as it's quite clear now that the issue I experience is due to some unfortunate combination of the versioning applied to the golang.zx2c4.com/wireguard module and my build environment.

joelpet commented 3 years ago

I have found that changing GOPROXY from "direct" to "https://proxy.golang.org,direct" makes the build succeed. I have also made the following observations:

❯ env --ignore-environment PATH=/usr/bin GOCACHE=(mktemp --directory) GOPATH=(mktemp --directory) GOPROXY="direct" go get golang.zx2c4.com/wireguard@v0.0.20201118
go get: golang.zx2c4.com/wireguard@v0.0.20201118: invalid version: unknown revision v0.0.20201118

❯ env --ignore-environment PATH=/usr/bin GOCACHE=(mktemp --directory) GOPATH=(mktemp --directory) GOPROXY="https://proxy.golang.org,direct" go get golang.zx2c4.com/wireguard@v0.0.20201118
go: downloading golang.zx2c4.com/wireguard v0.0.20201118
go: downloading golang.org/x/sys v0.0.0-20210309040221-94ec62e08169
go: downloading golang.org/x/net v0.0.0-20210226172049-e18ecbb05110
go: downloading golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83
go: warning: golang.zx2c4.com/wireguard@v0.0.20201118: retracted by module author
go: to switch to the latest unretracted version, run:
    go get golang.zx2c4.com/wireguard@latest
go get: upgraded golang.zx2c4.com/wireguard v0.0.0-20201118132417-da19db415a58 => v0.0.20201118

This seems to indicate that the required version v0.0.20201118 exists in the proxy.golang.org Go module cache, but is not available as a Git reference in the source repository.

mullvad / mullvadvpn-app

Building AUR package fails due to unknown revision of wireguard Go module dependency #2470

Issue report

Issue description

go.mod: