Go1.10 and the pkg cache

[keegancsmith]

The go build command now maintains a cache of recently built packages, separate from the installed packages in $GOROOT/pkg or $GOPATH/pkg. The effect of the cache should be to speed builds that do not explicitly install packages or when switching between different copies of source code (for example, when changing back and forth between different branches in a version control system). The old advice to add the -i flag for speed, as in go build -i or go test -i, is no longer necessary: builds run just as fast without -i. For more details, see go help cache.

From https://tip.golang.org/doc/go1.10#build

If I am not mistaken gocode uses the pkg dir to help complete. One specific example I know of is completing an import. It seems the pkg directory may be populated less often, so will not be as reliable for helping with completion. I am unsure if this will actually affect gocode in practice, just filing to open the discussion.

[nsf]

Yes, it sucks. With gocode you have to use "go install" anyway and that package cache feature is useless to you. Sadly there are no plans on my side to workaround it. I talked a bit about it a few times. A proper autocompletion service should use source files instead of package files. Somebody should make one. My enthusiasm for gocode is long gone.

I would suggest looking for alternatives. However I'm unaware of their state. I know there were few attempts of making "language server protocol" servers for Go. Have no idea how good they are. Sadly proper autocompletion tool of a kind should reimplement all language semantic analysis, because working with code that is being actively edited is a slightly different task that doing compilation or static analysis. I think I know how to make a proper service here, but... There's always but.. Maybe one day.

[keegancsmith]

I'm actually one of the maintainers of https://github.com/sourcegraph/go-langserver We had a nice contribution which embedded gocode for completion, so we just use gocode in that case. Yes it will require quite a lot of effort to do properly, our language server relies on (mostly) glueing stuff that already exists in go/x/tools and other great OSS projects to work. I believe there is work which hopefully is fruitful for a langserver from scratch at https://github.com/dominikh/go-tools/issues/136

I actually noticed this due to it failing one of the langserver regression tests on go1.10 See https://github.com/sourcegraph/go-langserver/pull/236/commits/db373e4e1f36fe9858c2ac7b36860214b3070c8c for more info.

[nsf]

Well ok, I'll have a closer look at this package cache feature (sooner or later). Will let you know then, maybe it's possible to make gocode work with that cached stuff as well.

[DisposaBoy]

FWIW, I've been using go1.10 and gocode with the autobuild option for about a month and haven't noticed any issues.

...but i regularly destroy the pkg directory and recompile things so YMMV

EDIT: also note that go install now has a new option -i so go install -i should restore the old behaviour.

[nsf]

I'm not going to do anything about it for now. Yeah, I know it sucks. Sorry about that. I really dislike this idea with package cache. It feels like another attempt to fix their broken package management system.

and the very concept of installed packages may disappear in a future release

Good luck with that, I'm out of here...

No, just kidding. Well, I talked about that in my "gocode rewrite" issue. The problem is that a proper tool should work with source files. Hence, there are two options for me:

1. Make gocode work with source files.
2. Rewrite it completely.

Problem: lack of motivation and free time. I definitely favor second option, but it requires way more time than the first one.

I don't know what else to say.

[zchee]

@nsf Hmm... The time to write gocode v2 came? If you really lack of motivation and free time, As I said before, I want to try to develop the alternative gocode.

What do you think about it? (I mean, the situation of Go core changes)

[nsf]

You don't need to ask me. If you can and want to make something that works well - do it. However, I may end up doing gocode v2 anyway one day.

I wrote gocode for myself and when time comes and gocode stops working for me, I'll look for alternatives. In case if I don't find one, I'll make whatever tool is necessary.

I think it's just a question about whether somebody can make something that works well and is willing to maintain it for years without expecting anything in return. So if you feel like doing it, do it. If not - whatever.

I'm not looking for help per se and won't help anyone either.

As for whether it's the time. Maybe, maybe not. I'm still using gb tool mainly (https://getgb.io), hence all that fuss around package management and go tool changes doesn't affect me at all.

[zchee]

@nsf thanks for describing your thought. Now Go core uses package cache, gocode doesn't support yet. But certainly gocode still fast completion engine for Go. Now didn't time comes and gocode stops working.

You don't need to ask me. If you can and want to make something that works well - do it. However, I may end up doing gocode v2 anyway one day.

Thanks, If I have time, will try to write PoC code. thanks :)

[myitcv]

@nsf - I think the question I asked in this thread is relevant here.

The one question that comes to mind is how this pipeline of tools will work together and whether the cache resolution step (which ultimately involves hashing of files) adds a significant delay to a gocode answer (because at the moment there is no resolution required). Feels like it won't be insignificant, and that therefore some sort of background ahead-of-time resolution might be required... but honestly I'm guessing without the numbers.

[nsf]

I said many times. Tools should work with source files only. The fact that "go/types" depends on compiled packages is a problem that needs to be solved. I don't care about "go/types" that much, don't use it, but other people who write tools might want to push that idea forward. Technically I haven't even looked into "go/types" code, last time I saw it, I remember it had some pluggable package resolvers of some kind. Maybe one can make it work with source files instead of compiled packages.

That's the ultimate solution for tools - just use source files. In particular it's even a requirement for things such as cross-package renaming.

[myitcv]

As I understand things, go/types does load sources files in the package of interest; it's imports that are loaded via go/types.Importer (which can follow whatever strategy you like, one of which is source file-based: https://godoc.org/golang.org/x/tools/go/loader).

In the fork https://github.com/mdempsky/gocode, @mdempsky made gocode go/types-based (ultimately using the go/loader importer)

Loading everything from source would surely be far too slow in general, no?

[nsf]

Using source files might be slower. But not by much.

[myitcv]

https://github.com/mdempsky/gocode/pull/26 is a proof-of-concept of vgo and (by definition since vgo requires Go 1.10) build-cache aware version of gocode.

As the PR mentions, it depends on https://go-review.googlesource.com/c/vgo/+/105855 which is very much WIP itself.

[metakeule]

Seems like the way to go is to use golang.org/x/tools/go/packages see:

https://github.com/golang/go/issues/24661

[thatjames]

I had this issue but managed to resolve it.

I can't explain what happened, but it appears as if GOPATH/pkg/destdir was not being updated when either go build -i or go install were being called. This is 99% likely due to the change in package caching.

Tanking the GOPATH/pkg/GOARCH/destdir directory and running go build -i in a loop recreated these objects and my autocomplete is working again.

Totally agree with @nsf about the tooling working directly with source code, this is just a workaround for the issue described here. Works well enough for me.

[myitcv]

https://github.com/stamblerre/gocode is the fork of gocode to use if you need/want Go modules support. This fork is based on go/packages, which uses the go build cache under the covers. This means that you in effect have source code-based completions, because the go build cache is rebuilt as required.

Otherwise https://github.com/mdempsky/gocode is the main fork; this is not based on go/packages and so can suffer from dependencies being out of date.

[thatjames]

@myitcv Ah, thank you :)