packer v1.4.5 fails to build from source when go modules are turned off

spavuluri commented 4 years ago

Overview of the Issue

Tried to build packer v1.4.5 from source. Using Go 1.13 and have modules turned off using export GO111MODULE=off. Build fails with error Stderr: builder/openstack/builder.hcl2spec.go:6:2: must be imported as github.com/gophercloud/gophercloud/openstack/imageservice/v2/images

Reproduction Steps

git clone packer repo
git checkout v1.4.5
export GO111MODULE=off
go generate ./...
./scripts/build.sh

Reports this error - Stderr: builder/openstack/builder.hcl2spec.go:6:2: must be imported as github.com/gophercloud/gophercloud/openstack/imageservice/v2/images

Packer version

From packer v1.4.5

Simplified Packer Buildfile

n/a

Operating system and Environment details

Amazon Linux 2

Log Fragments and crash.log files

n/a

spavuluri commented 4 years ago

Would like to note that this also happens when building using Go 1.12 with its default GO111MODULE.

nywilken commented 4 years ago

Hi @spavuluri thanks for opening up this issue. Go modules support is recommend, as the Packer project is using Go modules for managing project dependencies so you may run into trouble if all of the dependency don't already exist on the compiling system and you are working outside of the $GOPATH.

Is there a particular reason why you are setting GO111MODULE=off? If you are working within the$GOPATH have you tried running go get ./... before trying to build the project.

If you are looking to just to build a particular release of Packer for a particular version from source you may want to take a look at the compiling from source documentation.

For developing against the project we recommend the steps in the CONTRIBUTING guide.

Please let me know if this helps or if there is something about your setup that has not yet been captured in the issue. Cheers!

SwampDragons commented 4 years ago

I'm unable to reproduce unless I am not on the GOPATH, which is to be expected. Golang isn't gonna build properly outside the GOPATH if modules aren't enabled. Does your instance have its GOPATH set, and are you on it? If so, I just built v1.4.5 and it worked fine.

spavuluri commented 4 years ago

Hi @nywilken Thank you for your response.

We're working in an airgapped environment and hence relying on vendored dependencies for now. Supporting modules in our env. will take a bit longer. Does Packer support compiling on Go 1.12 now?

Looking at this specific error, it appears that the imageservice dependency in question is, in fact, provided in the vendor dir. However, mapstructure-to-hcl2 generates the import path as github.com/hashicorp/packer/vendor/github.com/gophercloud/gophercloud/openstack/imageservice/v2/images rather than github.com/gophercloud/gophercloud/openstack/imageservice/v2/images in the generated file builder/openstack/builder.hcl2spec.go.

@SwampDragons hi, thanks for looking into this. Yes, GOPATH is set and the project is on it. The problem seems to happen if you run go generate ./... before running the build itself. Please note step#4 in Reproduction Steps above. If I skip step#4, my build succeeds as well.

As noted in preceding sentence, its when go generate runs that the code generation by mapstructure-to-hcl2 seems to result in the problem import statement that then fails to build.

SwampDragons commented 4 years ago

Ah, my mistake was generating code before I'd set GO111MODULES=off. When I have modules turned off right away this does happen.

I'm not 100% clear on why this happens, but I don't think it's really Packer's fault; the code tagged as v1.4.5 builds fine; it's only when you modify the code from that tag by regenerating it that this happens.

The good news is that the mapstructure-to-hcl2 code isn't used yet. Those hcl2spec files are part of an enhancement we're hoping to roll out in v1.5.0, and they're currently only in the main branch so that they don't go stale as new changes to configs are made. That means that right now, if you're having unresolvable issues you can just delete any the files with the extension hcl2spec.go and Packer will run the same way as it would with the files present. I just ran rm ./builder/openstack/*.hcl2spec.go and reran the build script and it worked. Is this a viable workaround for you?

spavuluri commented 4 years ago

@SwampDragons

Thank you for that additional info about the hcl2spec files. Was able to devise a fix in our build hook to circumvent those files. And Packer now builds again in our setup.

While this is a viable workaround for us, I think folks building Packer with Go 1.12 will have issues. Unless they explicitly turn on modules in Go 1.12. That might affect Packer's backwards compatibility, although it only matters for folks building from source.

SwampDragons commented 4 years ago

Given that the issue only happens if you regenerate files, rather than just running the build script against the tagged release, I think the backwards compatibility effect is minimal.

As for the Golang version -- Packer is a small team, so we can currently only guarantee that Packer builds side-effect free on the most recent version of golang. We do keep our vendors directory up to date for people who don't want to build with modules, but I think we can only reasonably guarantee it for the tagged release.

azr commented 4 years ago

Hello there,

To give more insight; this happens in golang.org/x/tools/go/packages; here:

https://github.com/hashicorp/packer/blob/819329228a8dbf82908dc8ccf7c48ab53bfeb2f6/cmd/mapstructure-to-hcl2/mapstructure-to-hcl2.go#L81-L84

The packages package will load all types it finds and dependencies using the way the user wants; in your case it's using the vendor folder since go modules are off.

Later on when we output the autogenerated file we simply print the packages that we saw:

https://github.com/hashicorp/packer/blob/819329228a8dbf82908dc8ccf7c48ab53bfeb2f6/cmd/mapstructure-to-hcl2/mapstructure-to-hcl2.go#L327-L335

I tried removing the vendor folder and if all the dependencies are correctly present go generate will work correctly.

There are a few ways I can think of to address this :

make sure the usage of packages is correct; may be there's an option to display the 'real' pkg. or the real pkg should be taken from a different parsed object like the root pkg one ??
if not try to see if packages can be fixed to do this.
To quickly fix the symptoms of this, one could also simply check that pkg.Path contains a /vendor/ folder in mapstructure-to-hcl2.go and if so remove everything that's before that.

ghost commented 4 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / packer