cmd/go: go mod -vendor prunes non-package directories

ainar-g commented 6 years ago

What version of Go are you using (`go version`)?

Does this issue reproduce with the latest release?

go version devel +8a33045 Fri Jul 13 03:53:00 2018 +0000 linux/amd64

What operating system and processor architecture are you using (`go env`)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ainar/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/ainar/go"
GOPROXY=""
GORACE=""
GOROOT="/home/ainar/go/gotip"
GOTMPDIR=""
GOTOOLDIR="/home/ainar/go/gotip/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build865981061=/tmp/go-build -gno-record-gcc-switches"

What did you do?

#!/bin/sh
set -e
export GO111MODULE=on
mkdir foo
cd foo
cat <<EOF > main.go
package foo // import "foo"

import "github.com/gen2brain/aac-go/aacenc"

func f() { _ = aacenc.VoPidAacMdoule }
EOF
$GO mod -init
$GO mod -vendor
$GO build -getmode vendor

What did you expect to see?

Successful build.

What did you see instead?

# github.com/gen2brain/aac-go/aacenc
vendor/github.com/gen2brain/aac-go/aacenc/aacenc.go:4:19: fatal error: voAAC.h: No such file or directory
 //#include "voAAC.h"
                   ^
compilation terminated.

The problem is that go mod -vendor pruned the external/aacenc folder, which contains the C code needed to build this package.

dolmen commented 2 years ago

Looking https://github.com/golang/go/issues/26366#issuecomment-405683150, it seems that using a tree of C files will not work correctly.

But I still have to use code from a library that is organized as a tree. So I need to flatten that tree as files in my package directory.

Has anyone built a tool that will take a C file and resolve and inline all #include that are relative to a base directory? This is a subset of what a preprocessor does...

dolmen commented 2 years ago

Here is a workaround that may be useful for other projects:

For package github.com/dolmen-go/hid (which embeds C libraries libusb and hidapi) I've implemented an ad-hoc C preprocessor (triggered by go generate) that concatenates all required C files from the subdirectories and produces 3 .h files (one for each supported platform). The cgo source just includes the the local .h file corresponding to GOOS.

williamh commented 2 years ago

All,

I'm a packager on Gentoo, and I'm hitting this with https://git.sr.ht/~rjarry/aerc.git.

I am required to vendor to build packages under our package manager, and it doesn't look like that is going to change any time soon. Our package manager team sees allowing me to reach out to the network during a build as a massive security hole they don't want to open, and they tell me that having builds that are required to reach out to the network breaks building on unconnected systems.

So, what is the best way to deal with this issue?

Thanks,

William

williamh commented 2 years ago

I have packaged the vend tool from @nomad-software on Gentoo, thanks for writing it.

IMO this is a pretty significant issue since go mod vendor creates vendor directories that are unusable in some situations.

We will use vend for now, but please count me in as someone requesting that this discussion be restarted.

Thanks much for your time,

William

xuyang2 commented 2 years ago

@williamh Is it acceptable to install athens as a private proxy for Go modules? In this way, only athens needs to access the internet.

acestronautical commented 2 years ago

Why is this closed? This is still a major issue. Go dep was able to handle this with "unused-packages = false" which means many cannot and will not switch over to go mod.

bcmills commented 1 year ago

Why is this closed?

See https://github.com/golang/go/issues/26366#issuecomment-405683150.

golang / go