Problem with completion for external/vendores packages since 1.11

[dvcrn]

Since upgrading to go1.11, I noticed that gocode can no longer complete for external packages. It usually just displays invalid type.

• Using vim with vim-go and ncm2
• Switching to nsf/gocode is giving me autocomplete again
• I tried launching the server with additional flags in debug, but always the same issues
• Double checked packages are go installed

What version of Go are you using (go version)?

go version go1.11.1 darwin/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/d/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/d/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11.1/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11.1/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/1h/rlfwrw850qv10pfcnpfrkpxw0000gq/T/go-build866640166=/tmp/go-build -gno-record-gcc-switches -fno-common"

What is the debug output of gocode?

2018/10/26 13:44:44 -------------------------------------------------------
2018/10/26 13:44:44 Elapsed duration: 1.270449ms
2018/10/26 13:44:44 Offset: 0
2018/10/26 13:44:44 Number of candidates found: 1
2018/10/26 13:44:44 Candidates are:
2018/10/26 13:44:44   var msg *invalid type
2018/10/26 13:44:44 =======================================================

What (client-side) flags are you passing into gocode?

gocode -s -debug -source -unimpord-packages -builtin

[frzifus]

I've been watching this. go version go1.11 linux/amd64 using emacs with gocode, autocomplete does not work for external packages for example protobuf and some types are invalid 0f94242ccdb9fae06fce2c81cedafee64251e018:

works but emacs freezes completely every few minutes: c797e02c1ebebfdad6ba94ca8814a4a52270a5fc

[stamblerre]

@frzifus: your case sounds a bit different to me, as the difference between those commits is that I removed the default fallback behavior of importing from source, if no .a files are found for a package. Have you built your package and its dependencies? My guess is that the invalid types you see are packages that don't have a corresponding .a file in $GOPATH/pkg.

[stamblerre]

@dvcrn: What editor are you using/are you seeing this error in all editors?

[dvcrn]

@stamblerre yes! I tested it with fresh vscode, fresh sublime and fresh vim environment to rule out it's a problem with my editor config, and the behavior is the same.

[stamblerre]

@dvcrn: are you using a regular $GOPATH layout with go build? In the screenshot above, is that a vendored dependency?

[dvcrn]

@stamblerre yes and yes. I follow the normal $GOPATH layout and not use any of the new module stuff. It's also a clean mac install.

I tried to debug this a little bit more and found other strange things like:

With testutils being not a vendored package and the function itself just returning a random uint32

Or this

I set up a completely clean project, added the same dependencies and, go installed things and would you guessed it? It works! Then I checked out another project that uses the same deps, dep ensure and installed all the deps. Same problem, gocode can't complete a lot of things.

So there is probably something specific to my project setups that is messing with things. I tried the usual: wiping the project, checked it out again, re-downloaded vendors but the problem persists in all editors. These are projects I've been working on for the past year in the same GOPATH setup with the same editor.

If there's something I can try to debug this please let me know!

[dvcrn]

Thought of something else: I went back a few commits on gocode and wanted to see if I still have problems and yeah they were gone and gocode suggested just fine. Then tried to find the commit when it stops working

https://github.com/mdempsky/gocode/commit/4d2079acab8b3a6c0de16dd0f674c4960eba1453

2018/10/28 14:42:15 -------------------------------------------------------
2018/10/28 14:42:15 Elapsed duration: 15.12695ms
2018/10/28 14:42:15 Offset: 0
2018/10/28 14:42:15 Number of candidates found: 3
2018/10/28 14:42:15 Candidates are:
2018/10/28 14:42:15   package context
2018/10/28 14:42:15   var client *pubsub.Client
2018/10/28 14:42:15   var ctx context.Context
2018/10/28 14:42:15 =======================================================

https://github.com/mdempsky/gocode/commit/aef9c05e4dae07a5772c6c1a59b5f86ff821b828

2018/10/28 14:43:39 -------------------------------------------------------
2018/10/28 14:43:39 Elapsed duration: 3.832839ms
2018/10/28 14:43:39 Offset: 0
2018/10/28 14:43:39 Number of candidates found: 3
2018/10/28 14:43:39 Candidates are:
2018/10/28 14:43:39   package context
2018/10/28 14:43:39   var client invalid type
2018/10/28 14:43:39   var ctx context.Context
2018/10/28 14:43:39 =======================================================

So https://github.com/mdempsky/gocode/commit/aef9c05e4dae07a5772c6c1a59b5f86ff821b828 is for me when the invalid type stuff starts happening

[stamblerre]

It's not obvious to me why https://github.com/mdempsky/gocode/commit/aef9c05e4dae07a5772c6c1a59b5f86ff821b828 would cause the problem. It seems like a dependency package is not being resolved correctly, so if you are interested in debugging this further, I would try adding logs to print out in the Import function https://github.com/mdempsky/gocode/blob/5d5a61bb99f0bd165af975a7886446c0bcedc85d/internal/cache/importer.go#L87 to make sure that the package is able to be imported properly.

If you want to test if https://github.com/mdempsky/gocode/commit/aef9c05e4dae07a5772c6c1a59b5f86ff821b828 is the problem, I would try syncing to head, and then manually reverting that change to see if it fixes the problem. Just a reminder - make sure you run gocode close anytime you want to test a new version.

I'm sorry, I wish I could be more helpful in debugging this issue, but unfortunately I can't reproduce the problem :(

[jeroenrinzema]

I am experiencing the same issue where all variables and returned values from methods are marked as invalid types. This happens when a package is located outside the GOPATH and inside. Though reverting back to aef9c05 does not seem to resolve the issue. @stamblerre I will add some logs from the ImportFrom method in a bit.

[jeroenrinzema]

Logs from ImportedFrom:

fmt.Println("ImportFrom:", importPath, srcDir, mode)

ImportFrom: net/http /Users/jeroenrinzema/go/src/github.com/jeroenrinzema/commander-boilerplate/connector 0
ImportFrom: os /Users/jeroenrinzema/go/src/github.com/jeroenrinzema/commander-boilerplate/connector 0
ImportFrom: strings /Users/jeroenrinzema/go/src/github.com/jeroenrinzema/commander-boilerplate/connector 0
ImportFrom: github.com/gorilla/mux /Users/jeroenrinzema/go/src/github.com/jeroenrinzema/commander-boilerplate/connector 0
2018/10/29 19:47:28 Elapsed duration: 17.671715ms
2018/10/29 19:47:28 Offset: 0
2018/10/29 19:47:28 Number of candidates found: 2
2018/10/29 19:47:28 Candidates are:
2018/10/29 19:47:28   package rest 
2018/10/29 19:47:28   var router invalid type
2018/10/29 19:47:28 =======================================================

[stamblerre]

@jeroenrinzema: thanks for debugging this! can you try logging the if the returned "pkg" value is non-nil? Errors can occur silently here, but I will try to add a bit more debug logging soon.

[jeroenrinzema]

@stamblerre sure. I am logging all plausible return values. I noticed that the gocode is unable to locate the package even though the package exists.

ImportFrom: github.com/jeroenrinzema/commander /Users/jeroenrinzema/go/src/github.com/jeroenrinzema/commander-boilerplate/connector 0
ImportFrom error: stat : no such file or directory

Full log:

ImportFrom: net/http /Users/jeroenrinzema/go/src/github.com/jeroenrinzema/commander-boilerplate/connector 0
ImportFrom return values: package http ("net/http") <nil>
ImportFrom: os /Users/jeroenrinzema/go/src/github.com/jeroenrinzema/commander-boilerplate/connector 0
ImportFrom return values: package os ("os") <nil>
ImportFrom: strings /Users/jeroenrinzema/go/src/github.com/jeroenrinzema/commander-boilerplate/connector 0
ImportFrom return values: package strings ("strings") <nil>
ImportFrom: github.com/gorilla/mux /Users/jeroenrinzema/go/src/github.com/jeroenrinzema/commander-boilerplate/connector 0
ImportFrom error: stat : no such file or directory
ImportFrom: github.com/jeroenrinzema/commander /Users/jeroenrinzema/go/src/github.com/jeroenrinzema/commander-boilerplate/connector 0
ImportFrom error: stat : no such file or directory

[stamblerre]

Do the .a files for these packages exist in your $GOPATH/pkg directory?

[jeroenrinzema]

@stamblerre yes:

bash-3.2$ ls /Users/jeroenrinzema/go/src/github.com/jeroenrinzema/commander-boilerplate/connector
Dockerfile  README.md   controllers go.mod      go.sum      main.go     rest        vendor      websocket```

[stamblerre]

@jeroenrinzema: This version of gocode doesn't work with modules, so if you're using modules, you will need to use https://github.com/stamblerre/gocode.

Also, sorry I should've clarified, I actually meant if you could run ls /Users/jeroenrinzema/go/pkg/*/github.com/jeroenrinzema/commander-boilerplate. The * is just to sub in whatever architecture you're on.

[jeroenrinzema]

@stamblerre thank you i will take a look at that project

[dvcrn]

Did the same, debug output:

2018/10/30 10:34:10 -------------------------------------------------------
2018/10/30 10:34:10 Error parsing input file (outer block):
2018/10/30 10:34:10  /Users/d/go/src/github.com/dvcrn/myproject/pubsubclient/bar/client.go:27:4: expected selector or type assertion, found ';'
2018/10/30 10:34:10  /Users/d/go/src/github.com/dvcrn/myproject/pubsubclient/bar/client.go:29:2: expected ';', found err
ImportFrom: context /Users/d/go/src/github.com/dvcrn/myproject/pubsubclient/bar 0
Return values:  package context ("context")
ImportFrom: encoding/json /Users/d/go/src/github.com/dvcrn/myproject/pubsubclient/bar 0
Return values:  package json ("encoding/json")
ImportFrom: github.com/dvcrn/myproject/pubsub /Users/d/go/src/github.com/dvcrn/myproject/pubsubclient/bar 0
Return values:  stat : no such file or directory
ImportFrom: fmt /Users/d/go/src/github.com/dvcrn/myproject/pubsubclient/bar 0
Return values:  package fmt ("fmt")
ImportFrom: time /Users/d/go/src/github.com/dvcrn/myproject/pubsubclient/bar 0
Return values:  package time ("time")
2018/10/30 10:34:10 Elapsed duration: 10.456388ms
2018/10/30 10:34:10 Offset: 0
2018/10/30 10:34:10 Number of candidates found: 3
2018/10/30 10:34:10 Candidates are:
2018/10/30 10:34:10   func Send(ctx context.Context, message Message) error
2018/10/30 10:34:10   var publisher invalid type
2018/10/30 10:34:10   var topicName string
2018/10/30 10:34:10 =======================================================

(sanitized actual package names)

It can't import github.com/dvcrn/myproject/pubsub so I checked pkg/ for the a files

d@Davebook:~/go/pkg/darwin_amd64/github.com/dvcrn/myproject> ll
total 760
drwxr-xr-x  3 d  staff      96 Oct 29 11:32 auth
drwxr-xr-x  3 d  staff      96 Oct 29 11:32 bank
-rw-r--r--  1 d  staff  157978 Oct 29 11:33 database.a
-rw-r--r--  1 d  staff   35330 Oct 29 11:33 env.a
drwxr-xr-x  3 d  staff      96 Oct 29 15:22 grpcclient
-rw-r--r--  1 d  staff   34758 Oct 29 17:28 grpcclient.a
-rw-r--r--  1 d  staff   60320 Oct 29 11:33 logger.a
-rw-r--r--  1 d  staff   91802 Oct 30 10:39 pubsub.a     <-------
drwxr-xr-x  3 d  staff      96 Oct 29 11:24 pubsubclient

Same example as above:

2018/10/30 11:53:26 -------------------------------------------------------
ImportFrom: context /Users/d/go/src/github.com/dvcrn/myproject/src/foo/infrastructure/pubsub 0
Return values:  package context ("context")
ImportFrom: fmt /Users/d/go/src/github.com/dvcrn/myproject/src/foo/infrastructure/pubsub 0
Return values:  package fmt ("fmt")
ImportFrom: cloud.google.com/go/pubsub /Users/d/go/src/github.com/dvcrn/myproject/src/foo/infrastructure/pubsub 0
Return values:  stat : no such file or directory
2018/10/30 11:53:26 Elapsed duration: 3.310163ms
2018/10/30 11:53:26 Offset: 0
2018/10/30 11:53:26 Number of candidates found: 1
2018/10/30 11:53:26 Candidates are:
2018/10/30 11:53:26   var msg *invalid type
2018/10/30 11:53:26 =======================================================

pkg/

d@Davebook:~/go/pkg/darwin_amd64/github.com/dvcrn/myproject/vendor/cloud.google.com/go> ll
total 1200
drwxr-xr-x  3 d  staff      96 Oct 28 14:17 compute
-rw-r--r--  1 d  staff   95470 Oct 30 10:47 iam.a
drwxr-xr-x  4 d  staff     128 Oct 28 14:17 internal
drwxr-xr-x  4 d  staff     128 Oct 28 14:17 pubsub
-rw-r--r--  1 d  staff  514964 Oct 30 10:47 pubsub.a         <----

---

Project with the same deps that works:

2018/10/30 12:08:27 -------------------------------------------------------
2018/10/30 12:08:27 Error parsing input file (outer block):
2018/10/30 12:08:27  /Users/d/go/src/github.com/dvcrn/otherproject/main.go:37:4: expected selector or type assertion, found ';'
2018/10/30 12:08:27  /Users/d/go/src/github.com/dvcrn/otherproject/main.go:38:2: expected ';', found f
ImportFrom: cloud.google.com/go/pubsub /Users/d/go/src/github.com/dvcrn/otherproject 0
Return values:  package pubsub ("github.com/dvcrn/otherproject/vendor/cloud.google.com/go/pubsub")
2018/10/30 12:08:27 Elapsed duration: 5.414639ms
2018/10/30 12:08:27 Offset: 0
2018/10/30 12:08:27 Number of candidates found: 1
2018/10/30 12:08:27 Candidates are:
2018/10/30 12:08:27   var c *pubsub.Client
2018/10/30 12:08:27 =======================================================

pkg/

d@Davebook:~/go/pkg/darwin_amd64/github.com/dvcrn/otherproject/vendor/cloud.google.com/go> ll
total 1208
drwxr-xr-x  3 d  staff      96 Oct 28 14:20 compute
-rw-r--r--  1 d  staff   93562 Oct 30 11:54 iam.a
drwxr-xr-x  4 d  staff     128 Oct 28 14:20 internal
drwxr-xr-x  4 d  staff     128 Oct 28 14:20 pubsub
-rw-r--r--  1 d  staff  521434 Oct 30 11:54 pubsub.a    <----

I am very confused why this isn't working. I'll try and see if I can revert that commit and if it changes something

[dvcrn]

Good news - I reverted https://github.com/mdempsky/gocode/pull/71 onto latest master (fork here) and the autocomplete is working fine again. I'm guessing something in the cache implementation is messing with things?

If i can help debug this more to find the source of the problem, please let me know! For now I'll just use my own local version of gocode with the reverted change

[stamblerre]

Hm, seems like the function that's failing is gcexportdata.Find, since the stat fails on an empty filename. Do you have any symlinks or anything that could throw off that function (though it doesn't look like you do)? Do you possibly have an out-of-date copy of golang.org/x/tools? It could be worth trying go get -u github.com/mdempsky/gocode to update the dependencies. Otherwise, I'd suggest putting log statements inside of the gcexportdata.Find function.

[marwan-at-work]

having similar problems (using vscode). Non standard library packages are not being auto completed. I tried

$ go get -u github.com/mdempsky/gocode
$ gocode exit
$ gocode

and problem still there. This is only inside GOPATH since vscode doesn't use this fork for go modules.

[stamblerre]

@marwan-at-work: can you try putting in log statements like the ones above? I'd be curious to see if gcexportdata.Find is causing the problem for everyone.

[marwan-at-work]

@stamblerre I created a reproduction repo with the output you requested in the readme: https://github.com/marwan-at-work/gocodedebug

You can reproduce by trying to get an autocomplete from main.go by typing in sub. to try and auto complete the sub package.

[DisposaBoy]

@marwan-at-work I think there's a typo in your repro. You import gocodebug but the directory is named gocodedebug

[marwan-at-work]

@DisposaBoy the name of the github repo shouldn't matter (as long as you place it correctly in your GOPATH you'll be able to reproduce the issue) -- but thanks for the note, I'll update the code to reflect the same import path

[dvcrn]

@stamblerre I tried updating all the dependencies and re-installing gocode, but the problem persists. I will add log statements and report back!

[dvcrn]

I started debugging through what happens underneath and think I made some progress. I think it happens because I have a src/ directory in my project:

Debug from build.go when it tries to find the vendor folder:

/Users/d/go/src/github.com/dvcrn/myproject/src/foo/infrastructure/pubsub/vendor
/Users/d/go/src/github.com/dvcrn/myproject/src/foo/infrastructure/vendor
/Users/d/go/src/github.com/dvcrn/myproject/src/foo/vendor
/Users/d/go/src/github.com/dvcrn/myproject/src/vendor
/Users/d/go/src/github.com/dvcrn/myproject/src/foo/infrastructure/pubsub/vendor
/Users/d/go/src/github.com/dvcrn/myproject/src/foo/infrastructure/vendor
/Users/d/go/src/github.com/dvcrn/myproject/src/foo/vendor
/Users/d/go/src/github.com/dvcrn/myproject/src/vendor

The actual vendor/ is under /Users/d/go/src/github.com/dvcrn/myproject/vendor but it never gets to try that one.

So I compared both projects and inside the project that isn't working, srcDir that is passed to ImportFrom is /Users/d/go/src/github.com/dvcrn/myproject/src/foo/infrastructure/pubsub

In the project that is working, srcDir is simply /Users/d/go/src/github.com/dvcrn/otherproject

I wanted to debug this further but couldn't in the short time I had figure out where ImportFrom is being called from.

My thought: the src/ in the project is confusing whatever is constructing the srcDir

[edganiukov]

I have exactly the same issue. @dvcrn fyi, I dont have src directory in my project and the issue is still there.

[jcalvarado1965]

Not sure if this is related to the above. For me (go 1.11, Windows 10) auto-complete is not working for any external (under GOPATH or vendor) packages outside of GOROOT. Debugging gocode I traced the behavior to importer.joinPath in importer.go. This function is converting underscores to dashes in the GOOS_GOARCH part of the path. It is not clear to me why this is being done, but causes the search for binaries to fail, as the directory in my case has underscores (i.e. pkg/windows_amd64). Can anybody shed some light as to why this conversion is being done (maybe I'm missing something obvious)?

[parhamdoustdar]

I have the same issue on Mac OS with Go 1.11.1. The only types that are detected correctly are the ones defined in the same package, apparently.

As a lot of other people showed, the debug log also shows invalid type.

[stamblerre]

I just pushed https://github.com/mdempsky/gocode/commit/0af7a86943a6e0237c90f8aeb74a882e1862c898, which I hope will correct the problem. Please let me know if this works better.

[parhamdoustdar]

Thanks @stamblerre. No changes for me, though. Let's see if others have a different experience.

[parhamdoustdar]

One thing that might be contributing to this, at least for me, is that I don't see a new .a file under pkg/darwin_amd64 for my project. Running go install and go build doesn't produce any .a files. The files I have there are older, and hence don't contain my new changes. Now that I have deleted them, they don't reappear after running go install.

Has Go changed how it generates these files?

[stamblerre]

@parhamdoustdar: there should not be any changes in the way that .a files are produced. If you run go list -json on the name of your package, it should list the location of the .a file under "Target".

[DisposaBoy]

FWIW, the behaviour did change around go1.10 with the introduction of the build cache.

go install will not create the .a file inside GOPATH unless you install it explicitly or use go install -i.

That means that if you do go install pkg, pkg.a will be created, but deps.a will not (unless you used -i).

To see where the compiled package really is you need to use .Export i.e. go list -f '{{.Export}}' -export import/path instead of go list -f '{{.Target}}' import/path.

[philippfranke]

@stamblerre Thanks for looking into this issue.

I solved the problem in my environment by running:

go clean -cache
go build -i 

[stamblerre]

@philippfranke: thanks for letting me know - I think that building with -i is the correct solution for now. I will try to add a fix this week that will use the export data in the build cache.

[dvcrn]

fwiw, I've been using neomake with the following config to automatically go install with the -i flag and still have these issues.

let g:neomake_go_goinstall_maker = {
      \ 'exe': 'go',
      \ 'args': ['install', '-i'],
      \ 'append_file': 0,
      \ 'errorformat': '%f:%l:%c: %m',
      \ 'tempfile_enabled': 0,
      \ 'cwd': '%:h',
      \ }

Manually calling go install -i on specific packages that give me invalid type has no effect. What @philippfranke suggested didn't solve it for me neither.

I'm still running my fork without the cache pull request and it's working fine. Any other ideas what I could try?

[acroca]

I have the same issue and I've been trying installing previous commits of this repo and found out that I get the invalid type issue in all commits from 312a05504ee71febf5018e40fa8512c20df72d43.

[parhamdoustdar]

The -i flag fixed my issue.

This makes me think that maybe these are two different issues with the same symptoms?

[stamblerre]

I just hid the caching feature behind a flag since so many people had issues with it (https://github.com/mdempsky/gocode/commit/68f5472551baee41050cc975fa6e7cb7ad829024). If I have a chance to test it further I will try to work on resolving these issues and making caching the default again.