cmd/go: error isn't properly reported when running go list on cgo package with missing pkg-config

muirdm commented 4 years ago

What version of Go are you using (`go version`)?

$ go version
go version go1.13.3 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/muir/Library/Caches/go-build"
GOENV="/Users/muir/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/muir/projects//go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/_q/1xpb0k992ll803xsldpx0qyr0000gn/T/go-build252385107=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I opened a non-cgo package in our project. The package does not transitively depend on any cgo packages.

What did you expect to see?

I expected gopls to load the package with no issue.

What did you see instead?

gopls failed to create the view due to a go list "missing pkg-config" error.

LSP :: Error loading workspace folders (expected 1, got 0)
failed to load view for file:///Users/muir/projects/<project>/go/src: error loading packages: go [list -e -json -compiled=true -test=true -export=false -deps=true -find=false -- ./...]: exit status 2: # pkg-config --cflags  -- <bunch of libraries>
pkg-config: exec: "pkg-config": executable file not found in $PATH

Elsewhere, our project has a cgo package that uses the cgo pkg-config directive (e.g. // #cgo pkg-config: some-library). My laptop doesn't currently have pkg-config installed.

I assume the error comes from the go list ./... we do now when creating a view.

gopherbot commented 4 years ago

Thank you for filing a gopls issue! Please take a look at the Troubleshooting guide, and make sure that you have provided all of the relevant information here.

stamblerre commented 4 years ago

Thanks for reporting. This should probably be handled more gracefully in go/packages. Assigning to @matloob.

matloob commented 4 years ago

I can reproduce the underlying go list issue with the following layout:

foo/go.mod:

module foo

go1.13

foo/foo.go:

package foo

// #cgo pkg-config: bar
import "C"

output of go list -json -compiled . in directory foo:

{
    "Dir": "/Users/matloob/Desktop/foo",
    "ImportPath": "foo",
    "Name": "foo",
    "Root": "/Users/matloob/Desktop/foo",
    "Module": {
        "Path": "foo",
        "Main": true,
        "Dir": "/Users/matloob/Desktop/foo",
        "GoMod": "/Users/matloob/Desktop/foo/go.mod",
        "GoVersion": "1.13"
    },
    "Match": [
        "."
    ],
    "Stale": true,
    "StaleReason": "build ID mismatch",
    "CgoFiles": [
        "foo.go"
    ],
    "CgoPkgConfig": [
        "bar"
    ],
    "Imports": [
        "C",
        "unsafe",
        "runtime/cgo",
        "syscall"
    ],
    "Deps": [
        "errors",
        "internal/bytealg",
        "internal/cpu",
        "internal/oserror",
        "internal/race",
        "internal/reflectlite",
        "runtime",
        "runtime/cgo",
        "runtime/internal/atomic",
        "runtime/internal/math",
        "runtime/internal/sys",
        "sync",
        "sync/atomic",
        "syscall",
        "unsafe"
    ]
}

go list exits with an exit status of 2.

I'm going to send a cl to suppress the error in go/packages. There are still two problems in go list. The first is that -e isn't being respected. The second is that no error is being generated in the package output by go list.

gopherbot commented 4 years ago

Change https://golang.org/cl/210202 mentions this issue: go/packages: suppress go list -e error when pkg-config can't be run

matloob commented 4 years ago

I'm going to repurpose this bug to fix the issue in go list.

Note that even when the issue is fixed in go list, if go/packages or gopls are operating on a project that has a dependency on a cgo package that cgo can't successfully be run on, they won't have access to the full set of sources (which need to be generated by cgo). So they might still have problems for those projects.

Of course if none of the packages you're working on have a dependency on the offending package, things should work.

muirdm commented 4 years ago

Also note that even after installing pkg-config, go/packages still failed in a similar way because the C library wasn't installed.

matloob commented 4 years ago

What was the error it failed with? we're probably going to have to put another special case in go/packages to ignore that error...

muirdm commented 4 years ago

2019/12/06 11:16:40 failed to load snapshot: go [list -e -json -compiled=true -test=true -export=false -deps=true -find=false -- ./...]: exit status 2: # pkg-config --cflags  -- libavdevice libavformat libavfilter libavcodec libswscale libavutil libavdevice libavformat libavfilter libavcodec libswscale libavutil libavdevice libavformat libavfilter libavcodec libswscale libavutil libavdevice libavformat libavfilter libavcodec libswscale libavutil
Package libavdevice was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavdevice.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavdevice' found
Package libavformat was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavformat.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavformat' found
Package libavfilter was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavfilter.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavfilter' found
Package libavcodec was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavcodec.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavcodec' found
Package libswscale was not found in the pkg-config search path.
Perhaps you should add the directory containing `libswscale.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libswscale' found
Package libavutil was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavutil.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavutil' found
Package libavdevice was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavdevice.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavdevice' found
Package libavformat was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavformat.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavformat' found
Package libavfilter was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavfilter.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavfilter' found
Package libavcodec was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavcodec.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavcodec' found
Package libswscale was not found in the pkg-config search path.
Perhaps you should add the directory containing `libswscale.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libswscale' found
Package libavutil was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavutil.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavutil' found
Package libavdevice was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavdevice.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavdevice' found
Package libavformat was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavformat.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavformat' found
Package libavfilter was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavfilter.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavfilter' found
Package libavcodec was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavcodec.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavcodec' found
Package libswscale was not found in the pkg-config search path.
Perhaps you should add the directory containing `libswscale.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libswscale' found
Package libavutil was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavutil.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavutil' found
Package libavdevice was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavdevice.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavdevice' found
Package libavformat was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavformat.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavformat' found
Package libavfilter was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavfilter.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavfilter' found
Package libavcodec was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavcodec.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavcodec' found
Package libswscale was not found in the pkg-config search path.
Perhaps you should add the directory containing `libswscale.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libswscale' found
Package libavutil was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavutil.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavutil' found
pkg-config: exit status 1
# pkg-config --cflags  -- libavdevice libavformat libavfilter libavcodec libswscale libavutil libavdevice libavformat libavfilter libavcodec libswscale libavutil libavdevice libavformat libavfilter libavcodec libswscale libavutil libavdevice libavformat libavfilter libavcodec libswscale libavutil
Package libavdevice was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavdevice.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavdevice' found
Package libavformat was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavformat.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavformat' found
Package libavfilter was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavfilter.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavfilter' found
Package libavcodec was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavcodec.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavcodec' found
Package libswscale was not found in the pkg-config search path.
Perhaps you should add the directory containing `libswscale.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libswscale' found
Package libavutil was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavutil.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavutil' found
Package libavdevice was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavdevice.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavdevice' found
Package libavformat was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavformat.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavformat' found
Package libavfilter was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavfilter.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavfilter' found
Package libavcodec was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavcodec.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavcodec' found
Package libswscale was not found in the pkg-config search path.
Perhaps you should add the directory containing `libswscale.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libswscale' found
Package libavutil was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavutil.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavutil' found
Package libavdevice was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavdevice.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavdevice' found
Package libavformat was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavformat.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavformat' found
Package libavfilter was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavfilter.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavfilter' found
Package libavcodec was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavcodec.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavcodec' found
Package libswscale was not found in the pkg-config search path.
Perhaps you should add the directory containing `libswscale.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libswscale' found
Package libavutil was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavutil.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavutil' found
Package libavdevice was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavdevice.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavdevice' found
Package libavformat was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavformat.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavformat' found
Package libavfilter was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavfilter.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavfilter' found
Package libavcodec was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavcodec.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavcodec' found
Package libswscale was not found in the pkg-config search path.
Perhaps you should add the directory containing `libswscale.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libswscale' found
Package libavutil was not found in the pkg-config search path.
Perhaps you should add the directory containing `libavutil.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libavutil' found
pkg-config: exit status 1

    directory = file:///Users/muir/projects/<project>/go/src
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x16a69d5]

goroutine 136 [running]:
golang.org/x/tools/internal/lsp/cache.(*snapshot).View(0x0, 0xc00018cd80, 0xc0000ce300)
    /Users/muir/projects/tools/internal/lsp/cache/snapshot.go:61 +0x5
golang.org/x/tools/internal/lsp.(*Server).diagnoseSnapshot(0xc000278f60, 0x19cc4c0, 0x0)
    /Users/muir/projects/tools/internal/lsp/diagnostics.go:20 +0x69
created by golang.org/x/tools/internal/lsp.(*Server).addFolders
    /Users/muir/projects/tools/internal/lsp/general.go:173 +0x600

matloob commented 4 years ago

I'm abandoning the change. We've decided that the cost of the change is greater than the benefit. Because it's impossible to determine which package the error belongs to, we'd have to drop the error (or add errors to all packages) to work around it.

In general it's hard for go/packages to work correctly for code that doesn't build.

muirdm commented 4 years ago

When "go list" produces the cgo error and exits uncleanly it still also dumps what seems to be the normal JSON results (minus the cgo generated files). What would happen if we ignored the error and continued using the results?

If that is not tenable, is there some way we can at least make gopls able to start in the case of unbuildable cgo package?

stamblerre commented 4 years ago

Did CL 209420 help with this at all, or does gopls still fail in this case? If it does, I can add some workarounds in gopls to handle it.

muirdm commented 4 years ago

I uninstalled ffmpeg and still got the above panic (on master). Should be easy to reproduce by adding something like this to a file and then restarting gopls:

// #cgo pkg-config: some-library
import "C"

stamblerre commented 4 years ago

Ah, I just fixed that panic in CL 210215.

muirdm commented 4 years ago

Hot dog! Yes, seems to start up fine now.

stamblerre commented 4 years ago

Perfect! So it sounds like we will wait for this fix to be implemented in go list.

jayconrod commented 4 years ago

I think this is one instance of a broader problem: execution errors (from running the compiler, linker, or other tools) are not connected back to go list -e output.

Fixing that would be too complicated for 1.14 at this point in the freeze. We should consider it for 1.15 though.

ianlancetaylor commented 4 years ago

Moving to backlog.

golang / go