search

Homebrew / legacy-homebrew

💀 The former home of Homebrew/homebrew (deprecated)
https://brew.sh
26.96k stars 11.35k forks source link

Packer missing amazon builders #48293

Closed chiefy closed 8 years ago

chiefy commented 8 years ago

Did a fresh install of Packer, and it's mising the amazon related builders:

usr/local/Cellar/packer/0.8.6_1/bin master
❯ ls -lah
total 399M
drwxr-xr-x 38 ext.cnajewicz admin 1.3K Aug 21 22:54 ./
drwxr-xr-x  4 ext.cnajewicz admin  136 Jan 20 15:38 ../
-r-xr-xr-x  1 ext.cnajewicz admin  12M Aug 21 22:54 packer*
-r-xr-xr-x  1 ext.cnajewicz admin  13M Aug 21 22:54 packer-builder-digitalocean*
-r-xr-xr-x  1 ext.cnajewicz admin  13M Aug 21 22:54 packer-builder-docker*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-builder-file*
-r-xr-xr-x  1 ext.cnajewicz admin  15M Aug 21 22:54 packer-builder-googlecompute*
-r-xr-xr-x  1 ext.cnajewicz admin  12M Aug 21 22:54 packer-builder-null*
-r-xr-xr-x  1 ext.cnajewicz admin  13M Aug 21 22:54 packer-builder-openstack*
-r-xr-xr-x  1 ext.cnajewicz admin  13M Aug 21 22:54 packer-builder-parallels-iso*
-r-xr-xr-x  1 ext.cnajewicz admin  13M Aug 21 22:54 packer-builder-parallels-pvm*
-r-xr-xr-x  1 ext.cnajewicz admin  13M Aug 21 22:54 packer-builder-qemu*
-r-xr-xr-x  1 ext.cnajewicz admin  13M Aug 21 22:54 packer-builder-virtualbox-iso*
-r-xr-xr-x  1 ext.cnajewicz admin  13M Aug 21 22:54 packer-builder-virtualbox-ovf*
-r-xr-xr-x  1 ext.cnajewicz admin  13M Aug 21 22:54 packer-builder-vmware-iso*
-r-xr-xr-x  1 ext.cnajewicz admin  13M Aug 21 22:54 packer-builder-vmware-vmx*
-r-xr-xr-x  1 ext.cnajewicz admin  10M Aug 21 22:54 packer-post-processor-artifice*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-post-processor-atlas*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-post-processor-compress*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-post-processor-docker-import*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-post-processor-docker-push*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-post-processor-docker-save*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-post-processor-docker-tag*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-post-processor-vagrant*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-post-processor-vagrant-cloud*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-post-processor-vsphere*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-ansible-local*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-chef-client*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-chef-solo*
-r-xr-xr-x  1 ext.cnajewicz admin  10M Aug 21 22:54 packer-provisioner-file*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-powershell*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-puppet-masterless*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-puppet-server*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-salt-masterless*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-shell*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-shell-local*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-windows-restart*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-windows-shell*
DomT4 commented 8 years ago

This is interesting, Thanks for the report!

Taking another look at the build process it's failing to build those elements, but not regarding that failure as fatal.

3 errors occurred:
--> darwin/amd64 error: exit status 2
Stderr: # github.com/mitchellh/packer/builder/amazon/common
builder/amazon/common/artifact.go:75: cannot use regionConfig (type *aws.Config) as type client.ConfigProvider in argument to ec2.New:
    *aws.Config does not implement client.ConfigProvider (missing ClientConfig method)
builder/amazon/common/step_ami_region_copy.go:90: cannot use awsConfig (type *aws.Config) as type client.ConfigProvider in argument to ec2.New:
    *aws.Config does not implement client.ConfigProvider (missing ClientConfig method)
builder/amazon/common/step_create_tags.go:39: cannot use aws.Config literal (type *aws.Config) as type client.ConfigProvider in argument to ec2.New:
    *aws.Config does not implement client.ConfigProvider (missing ClientConfig method)
builder/amazon/common/step_modify_ami_attributes.go:93: cannot use aws.Config literal (type *aws.Config) as type client.ConfigProvider in argument to ec2.New:
    *aws.Config does not implement client.ConfigProvider (missing ClientConfig method)

--> darwin/amd64 error: exit status 2
Stderr: # github.com/mitchellh/packer/builder/amazon/common
builder/amazon/common/artifact.go:75: cannot use regionConfig (type *aws.Config) as type client.ConfigProvider in argument to ec2.New:
    *aws.Config does not implement client.ConfigProvider (missing ClientConfig method)
builder/amazon/common/step_ami_region_copy.go:90: cannot use awsConfig (type *aws.Config) as type client.ConfigProvider in argument to ec2.New:
    *aws.Config does not implement client.ConfigProvider (missing ClientConfig method)
builder/amazon/common/step_create_tags.go:39: cannot use aws.Config literal (type *aws.Config) as type client.ConfigProvider in argument to ec2.New:
    *aws.Config does not implement client.ConfigProvider (missing ClientConfig method)
builder/amazon/common/step_modify_ami_attributes.go:93: cannot use aws.Config literal (type *aws.Config) as type client.ConfigProvider in argument to ec2.New:
    *aws.Config does not implement client.ConfigProvider (missing ClientConfig method)

--> darwin/amd64 error: exit status 2
Stderr: # github.com/mitchellh/packer/builder/amazon/common
builder/amazon/common/artifact.go:75: cannot use regionConfig (type *aws.Config) as type client.ConfigProvider in argument to ec2.New:
    *aws.Config does not implement client.ConfigProvider (missing ClientConfig method)
builder/amazon/common/step_ami_region_copy.go:90: cannot use awsConfig (type *aws.Config) as type client.ConfigProvider in argument to ec2.New:
    *aws.Config does not implement client.ConfigProvider (missing ClientConfig method)
builder/amazon/common/step_create_tags.go:39: cannot use aws.Config literal (type *aws.Config) as type client.ConfigProvider in argument to ec2.New:
    *aws.Config does not implement client.ConfigProvider (missing ClientConfig method)
builder/amazon/common/step_modify_ami_attributes.go:93: cannot use aws.Config literal (type *aws.Config) as type client.ConfigProvider in argument to ec2.New:
    *aws.Config does not implement client.ConfigProvider (missing ClientConfig method)

==> Copying binaries for this platform...

The most obvious thing that has changed in terms of build environment between the 0.8.6 and 0.8.6_1 builds is that Go has received several updates in that time. Go would have been version 1.5 when this was built originally, compared to 1.5.3 a couple days back. I wonder if there are some subtle incompatibilities there.

As you can see in this commit I didn't change anything about the build itself there; the only changes were to bump the revision in our post Go-1.5.3 security fix wide rebuild and style nits.

There's an upstream bug report here as well.

chiefy commented 8 years ago

@DomT4 weird! Thanks for the update. If I have some time this weekend, I might try to look into it.

DomT4 commented 8 years ago

Thanks! Apologies it took me a couple of days to peek at this.

I've asked in the upstream bug report whether they're expecting to build against a specific Go release/are aware of any incompatibilities with the newer Go versions. That seems to be the most obvious potential point of failure to me.

It's possibly worth rolling Go back a release or two locally and building again to see whether the error resolves. It may not be tonight at this point but I'll try and poke around ASAP with the earlier versions and this formula to see if any progress can be made there.

For the benefit of anyone else who wants to try the same thing, please note Go 1.5 to 1.5.2 now have a publicly known, fairly severe security problem and if you downgrade to test please bear that in mind.

mistydemeo commented 8 years ago

We have the dependencies pinned to specific releases, but I'm wondering if there's a possibility we're missing some of the required dependencies and they're being fetched at buildtime. That would result in the current versions being pulled, which would break in this scenario.

DomT4 commented 8 years ago

We have the dependencies pinned to specific releases, but I'm wondering if there's a possibility we're missing some of the required dependencies and they're being fetched at buildtime. 

==> make bin
go get -v -d ./...
github.com/aws/aws-sdk-go (download)
github.com/go-ini/ini (download)
github.com/jmespath/go-jmespath (download)
github.com/dylanmei/winrmtest (download)
github.com/masterzen/winrm (download)
github.com/masterzen/simplexml (download)
github.com/masterzen/xmlpath (download)
github.com/satori/go.uuid (download)
github.com/nu7hatch/gouuid (download)
github.com/packer-community/winrmcp (download)
github.com/dylanmei/iso8601 (download)
github.com/digitalocean/godo (download)
github.com/google/go-querystring (download)
github.com/tent/http-link-go (download)
Fetching https://gopkg.in/fsnotify.v0?go-get=1
Parsing meta tags from https://gopkg.in/fsnotify.v0?go-get=1 (status code 200)
get "gopkg.in/fsnotify.v0": found meta tag main.metaImport{Prefix:"gopkg.in/fsnotify.v0", VCS:"git", RepoRoot:"https://gopkg.in/fsnotify.v0"} at https://gopkg.in/fsnotify.v0?go-get=1
gopkg.in/fsnotify.v0 (download)
github.com/rackspace/gophercloud (download)
github.com/klauspost/pgzip (download)
github.com/klauspost/compress (download)
github.com/klauspost/cpuid (download)
github.com/klauspost/crc32 (download)
github.com/pierrec/lz4 (download)
github.com/pierrec/xxHash (download)
==> Getting dependencies...
==> Removing old directory...

My download speed rises severely during the build, even with the laptop doing nothing else, so guess so.

mistydemeo commented 8 years ago

Yeah, looks like we missed a lot. This was made using homebrew-go-resources, which IIRC is known to miss dependencies.

DomT4 commented 8 years ago

Looks like because they aren't pinned upstream at all cloning the repo into the GOPATH, checking out the commit from the release, running make updatedeps and gdm brew simply delivers the commit hashes currently at the tip of the master branches for the deps.

Looks like this may well have to be done manually for each missing dependency unless you have any clever solutions I'm overlooking/missing entirely? Please say you do :smile:.

mistydemeo commented 8 years ago

gdm brew simply delivers the commit hashes currently at the tip of the master branches for the deps.

Yeah, this'd have been fine if we had done this at the time packer was new, but our options are more limited now. I wasn't aware of the homewbrew-go-resources issue at the time. :/

DomT4 commented 8 years ago

Garg. I manually went back and found the right commits for the upstream dependencies at the time the packer tag was cut and still hitting the same build failure as further up.

DomT4 commented 8 years ago

And it doesn't build properly against Go 1.4 or outside HOMEBREW_SANDBOX either :sweat:.

DomT4 commented 8 years ago

Ugh, I've just realised Atom's tab autocomplete has been putting revison instead of revision. Checking again.

DomT4 commented 8 years ago

That worked. Filed a PR at https://github.com/Homebrew/homebrew/pull/48393 if anyone wants to test, but the Amazon builders are there locally this time.

ln -s ../Cellar/packer/0.8.6_1/bin/packer-builder-amazon-chroot packer-builder-amazon-chroot
ln -s ../Cellar/packer/0.8.6_1/bin/packer-builder-amazon-ebs packer-builder-amazon-ebs
ln -s ../Cellar/packer/0.8.6_1/bin/packer-builder-amazon-instance packer-builder-amazon-instance
ln -s ../Cellar/packer/0.8.6_1/bin/packer-builder-digitalocean packer-builder-digitalocean
[snip]
chiefy commented 8 years ago 
❯ brew install packer
==> Downloading https://homebrew.bintray.com/bottles/packer-0.8.6_2.mavericks.bottle.tar.gz
######################################################################## 100.0%
==> Pouring packer-0.8.6_2.mavericks.bottle.tar.gz
🍺  /usr/local/Cellar/packer/0.8.6_2: 40 files, 446.7M

/usr/local/bin master 7m 12s
❯ ls -lah /usr/local/Cellar/packer/0.8.6_2/bin
total 447M
drwxr-xr-x 41 ext.cnajewicz admin 1.4K Aug 21 22:54 ./
drwxr-xr-x  4 ext.cnajewicz admin  136 Jan 23 10:21 ../
-r-xr-xr-x  1 ext.cnajewicz admin  12M Aug 21 22:54 packer*
-r-xr-xr-x  1 ext.cnajewicz admin  15M Aug 21 22:54 packer-builder-amazon-chroot*
-r-xr-xr-x  1 ext.cnajewicz admin  17M Aug 21 22:54 packer-builder-amazon-ebs*
-r-xr-xr-x  1 ext.cnajewicz admin  17M Aug 21 22:54 packer-builder-amazon-instance*
-r-xr-xr-x  1 ext.cnajewicz admin  13M Aug 21 22:54 packer-builder-digitalocean*
-r-xr-xr-x  1 ext.cnajewicz admin  13M Aug 21 22:54 packer-builder-docker*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-builder-file*
-r-xr-xr-x  1 ext.cnajewicz admin  15M Aug 21 22:54 packer-builder-googlecompute*
-r-xr-xr-x  1 ext.cnajewicz admin  12M Aug 21 22:54 packer-builder-null*
-r-xr-xr-x  1 ext.cnajewicz admin  13M Aug 21 22:54 packer-builder-openstack*
-r-xr-xr-x  1 ext.cnajewicz admin  13M Aug 21 22:54 packer-builder-parallels-iso*
-r-xr-xr-x  1 ext.cnajewicz admin  13M Aug 21 22:54 packer-builder-parallels-pvm*
-r-xr-xr-x  1 ext.cnajewicz admin  13M Aug 21 22:54 packer-builder-qemu*
-r-xr-xr-x  1 ext.cnajewicz admin  13M Aug 21 22:54 packer-builder-virtualbox-iso*
-r-xr-xr-x  1 ext.cnajewicz admin  13M Aug 21 22:54 packer-builder-virtualbox-ovf*
-r-xr-xr-x  1 ext.cnajewicz admin  13M Aug 21 22:54 packer-builder-vmware-iso*
-r-xr-xr-x  1 ext.cnajewicz admin  13M Aug 21 22:54 packer-builder-vmware-vmx*
-r-xr-xr-x  1 ext.cnajewicz admin  10M Aug 21 22:54 packer-post-processor-artifice*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-post-processor-atlas*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-post-processor-compress*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-post-processor-docker-import*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-post-processor-docker-push*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-post-processor-docker-save*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-post-processor-docker-tag*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-post-processor-vagrant*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-post-processor-vagrant-cloud*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-post-processor-vsphere*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-ansible-local*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-chef-client*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-chef-solo*
-r-xr-xr-x  1 ext.cnajewicz admin  10M Aug 21 22:54 packer-provisioner-file*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-powershell*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-puppet-masterless*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-puppet-server*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-salt-masterless*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-shell*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-shell-local*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-windows-restart*
-r-xr-xr-x  1 ext.cnajewicz admin  11M Aug 21 22:54 packer-provisioner-windows-shell*

Thanks! @DomT4