Flaky buildpack detection

[jesseshieh]

When I run docker run --rm -v /tmp/app:/tmp/app gliderlabs/herokuish /bin/herokuish buildpack build, the buildpack detected can be flaky. Any help is appreciated!

My app has a .buildpack file which means the 00_buildpack-multi should be detected. My app also has a package.json file, which means 02_buildpack-nodejs should also be detected, but because the buildpacks are ordered, I expect 00_buildpack-multi to be selected.

When I run the build, I get 02_buildpack-nodejs roughly 95% of the time and 00_buildpack-multi 5% of the time. 00_buildpack-multi is not even detected.

I did some testing and found that whatever buildpack is listed first gets randomly skipped and I'm not sure why. I manually deleted all the buildpacks, and installed 3 identical buildpacks. Sometimes herokuish reports 3 buildpacks detected, sometimes 2 buildpacks detected and it's always the first one that is skipped. I wasn't able to modify herokuish, but it almost seems as if buildpack-list() is not getting the first buildpack sometimes.

More details:

I'm trying to build an elixir app. I use 00_buildpack-multi so that I can use https://github.com/HashNuke/heroku-buildpack-elixir and https://github.com/gjaldon/heroku-buildpack-phoenix-static, but phoenix creates a package.json file to compile static assets.

What I'm seeing in my shell

~/tmp/dummy-app$ ls -a
.  ..  .buildpacks  package.json
~/tmp/dummy-app$ sudo docker run --rm -v $PWD:/tmp/app gliderlabs/herokuish /bin/herokuish buildpack build
-----> Node.js app detected
~/tmp/dummy-app$ sudo docker run --rm -v $PWD:/tmp/app gliderlabs/herokuish /bin/herokuish buildpack build
-----> Node.js app detected
~/tmp/dummy-app$ sudo docker run --rm -v $PWD:/tmp/app gliderlabs/herokuish /bin/herokuish buildpack build
-----> Node.js app detected
~/tmp/dummy-app$ sudo docker run --rm -v $PWD:/tmp/app gliderlabs/herokuish /bin/herokuish buildpack build              
-----> Warning: Multiple default buildpacks reported the ability to handle this app. The first buildpack in the list below will be used.
       Detected buildpacks: multi nodejs
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/HashNuke/heroku-buildpack-elixir

Versions

~/tmp/dummy-app$ sudo docker run --rm -v $PWD:/tmp/app gliderlabs/herokuish /bin/herokuish version
herokuish: 0.3.22
buildpacks:
  heroku-buildpack-multi     v1.0.0
  heroku-buildpack-ruby      v148
  heroku-buildpack-nodejs    v91
  heroku-buildpack-clojure   v75
  heroku-buildpack-python    v85
  heroku-buildpack-java      v48
  heroku-buildpack-gradle    v18
  heroku-buildpack-grails    v21
  heroku-buildpack-scala     v72
  heroku-buildpack-play      v26
  heroku-buildpack-php       v114
  heroku-buildpack-go        v52
  heroku-buildpack-erlang    fa17af9
  buildpack-nginx            v6
~/tmp/dummy-app$ uname -a
Linux foo 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
~/tmp/dummy-app$ sudo docker version
Client:
 Version:      1.13.0
 API version:  1.25
 Go version:   go1.7.3
 Git commit:   49bf474
 Built:        Tue Jan 17 09:58:26 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.0
 API version:  1.25 (minimum version 1.12)
 Go version:   go1.7.3
 Git commit:   49bf474
 Built:        Tue Jan 17 09:58:26 2017
 OS/Arch:      linux/amd64
 Experimental: false
~/tmp/dummy-app$ sudo docker images
REPOSITORY             TAG                 IMAGE ID            CREATED             SIZE
gliderlabs/herokuish   latest              14a0c05d8c2d        29 hours ago        1.4 GB

[josegonzalez]

Can you duplicate the flaky builds with TRACE on?

[michaelshobbs]

@jesseshieh can we also have the contents of your .buildpacks and package.json?

[jesseshieh]

Here is the full trace output. I ran the build twice. The first time 00_buildpack-multi was detected, the second time it was not. https://gist.github.com/anonymous/668af6455f71b56deb8f49408756080a

[jesseshieh]

Here are the contents of the files.

user@foo:~/app$ cat .buildpacks 
https://github.com/HashNuke/heroku-buildpack-elixir
https://github.com/gjaldon/heroku-buildpack-phoenix-static
user@foo:~/app$ cat package.json 
user@foo:~/app$ 

[michaelshobbs]

I can't make this happen. I built an image based on the v0.3.22 tag and composed an app dir like yours. Any other bits that could be different?

root@dokku:~/testapp# ls -a
.  ..  .buildpacks  package.json
root@dokku:~/testapp# cat .buildpacks
https://github.com/HashNuke/heroku-buildpack-elixir
https://github.com/gjaldon/heroku-buildpack-phoenix-static
root@dokku:~/testapp# cat package.json
root@dokku:~/testapp#
root@dokku:~/testapp# docker run --rm -v $PWD:/tmp/app herokuish:dev /bin/herokuish version
herokuish: dev
buildpacks:
  heroku-buildpack-multi     v1.0.0
  heroku-buildpack-ruby      v148
  heroku-buildpack-nodejs    v91
  heroku-buildpack-clojure   v75
  heroku-buildpack-python    v85
  heroku-buildpack-java      v48
  heroku-buildpack-gradle    v18
  heroku-buildpack-grails    v21
  heroku-buildpack-scala     v72
  heroku-buildpack-play      v26
  heroku-buildpack-php       v114
  heroku-buildpack-go        v52
  heroku-buildpack-erlang    fa17af9
  buildpack-nginx            v6
root@dokku:~/testapp# docker version
Client:
 Version:      1.12.5
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   7392c3b
 Built:        Fri Dec 16 02:30:42 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.5
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   7392c3b
 Built:        Fri Dec 16 02:30:42 2016
 OS/Arch:      linux/amd64
root@dokku:~/testapp# for i in $(seq 1 10); do docker run --rm -v $PWD:/tmp/app herokuish:dev /bin/herokuish buildpack build; done
-----> Warning: Multiple default buildpacks reported the ability to handle this app. The first buildpack in the list below will be used.
       Detected buildpacks: multi nodejs
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/HashNuke/heroku-buildpack-elixir
-----> Warning: Multiple default buildpacks reported the ability to handle this app. The first buildpack in the list below will be used.
       Detected buildpacks: multi nodejs
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/HashNuke/heroku-buildpack-elixir
-----> Warning: Multiple default buildpacks reported the ability to handle this app. The first buildpack in the list below will be used.
       Detected buildpacks: multi nodejs
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/HashNuke/heroku-buildpack-elixir
-----> Warning: Multiple default buildpacks reported the ability to handle this app. The first buildpack in the list below will be used.
       Detected buildpacks: multi nodejs
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/HashNuke/heroku-buildpack-elixir
-----> Warning: Multiple default buildpacks reported the ability to handle this app. The first buildpack in the list below will be used.
       Detected buildpacks: multi nodejs
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/HashNuke/heroku-buildpack-elixir
-----> Warning: Multiple default buildpacks reported the ability to handle this app. The first buildpack in the list below will be used.
       Detected buildpacks: multi nodejs
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/HashNuke/heroku-buildpack-elixir
-----> Warning: Multiple default buildpacks reported the ability to handle this app. The first buildpack in the list below will be used.
       Detected buildpacks: multi nodejs
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/HashNuke/heroku-buildpack-elixir
-----> Warning: Multiple default buildpacks reported the ability to handle this app. The first buildpack in the list below will be used.
       Detected buildpacks: multi nodejs
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/HashNuke/heroku-buildpack-elixir
-----> Warning: Multiple default buildpacks reported the ability to handle this app. The first buildpack in the list below will be used.
       Detected buildpacks: multi nodejs
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/HashNuke/heroku-buildpack-elixir
-----> Warning: Multiple default buildpacks reported the ability to handle this app. The first buildpack in the list below will be used.
       Detected buildpacks: multi nodejs
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/HashNuke/heroku-buildpack-elixir

[kevinob11]

I was able to recreate the issue using this repo:

https://github.com/praece/dokku-multi-test

I put the output of my deploys in the dokku_trace_output folder, there is one example of it properly detecting multiple buildpacks and one where it just uses nodejs. I didn't get full trace output in the files for some reason, I'll try to get that done tonight.

[michaelshobbs]

@kevinob11 thanks! I'll take a look ASAP.

[michaelshobbs]

I still can't get this to happen locally. Can either of you pull down the latest herokuish (v0.3.25) and make it happen again? The more detail about your setup the better.

[kevinob11]

Sure, I've never used herokuish outside of dokku, but the docs look pretty straightforward. I'll give it a shot asap, but it will likely be this weekend.

Good explanation of my environment is available here: https://github.com/dokku/dokku/issues/2551#issue-199411854

[jesseshieh]

One weird thing I've noticed recently: in my setup, I use packer to generate a Google Compute Platform image based on ubuntu-1604-lts. Sometimes, when I generate an image (image A), the flakiness is there and no matter how many VMs I launch with image A, it's always flaky. Then, I'll generate another image (image B), and every new VM I launch with image B will not be flaky. So the flakiness seems to be baked into the image somehow. The thing that confuses me is that image A and image B are constructed from the exact same packer config. Here is a redacted excerpt from my packer file.

{
  "builders": [{
    "image_family": "foo",
    "image_name": "foo-{{timestamp}}",
    "type": "googlecompute",
    "project_id": "foo",
    "source_image_family": "ubuntu-1604-lts",
    "zone": "us-central1-a",
    "ssh_username": "foo",
    "scopes": [
      "https://www.googleapis.com/auth/devstorage.read_write"
    ]
  }],
  "provisioners": [
    {
      "type": "shell",
      "inline": [
        "sudo apt-get update",
        "# docker",
        "sudo apt-get install -y apt-transport-https ca-certificates",
        "sudo apt-key adv --keyserver hkp://ha.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D",
        "echo 'deb https://apt.dockerproject.org/repo ubuntu-xenial main' | sudo tee /etc/apt/sources.list.d/docker.list",
        "sudo apt-get update",
        "sudo apt-get install -y linux-image-extra-$(uname -r) linux-image-extra-virtual",
        "sudo apt-get install -y docker-engine",
        "sudo usermod -aG docker www-data",
      ]
    }
  ]
}

[x-ji]

I just had this issue when trying to use Gigalixir. Took a while to figure out that my app was actually built as a Node.js app (since it has a package.json at the root)... No wonder it didn't run! The error messages from the logs were not very helpful. I just took it for granted that since I have a rel/config.exs, it will be detected as an Elixir app with Distillery automatically.

I had to manually add a .buildpack file to make it work correctly.

@jesseshieh Maybe the documentation could be updated to warn people of such possibilities.

[jesseshieh]

@x-ji thanks for the feedback! That's a good idea. I'll put it in the docs somewhere. Also happy to accept pull requests if you know a good place to put it. https://github.com/gigalixir/gigalixir-cli/blob/master/docs/source/main.rst

[x-ji]

👌 Didn't know the docs are open source. I can give it a try.

[x-ji]

@jesseshieh Added relevant information in the docs https://github.com/gigalixir/gigalixir-cli/pull/48