Using Heroku, fails when composing from Docker

[dja]

Describe the bug A clear and concise description of what the bug is. Deploying with Heroku Container Registry fails to start

To reproduce Following Heroku instructions to deploy using the Docker container

$ docker tag launchdarkly/ld-relay registry.heroku.com/<app>/web
$ docker push registry.heroku.com/<app>/web

Expected behavior It should start the docker container successfully and run the Relay Proxy, using the Environment Variables specified in the Heroku app's config.

Logs If applicable, add any log output related to your problem.

2021-04-25T03:39:36.839799+00:00 heroku[web.1]: Starting process with command `/usr/bin/ldr --config /ldr/ld-relay.conf --allow-missing-file --from-env`
2021-04-25T03:39:39.636712+00:00 app[web.1]: Error: Exec format error
2021-04-25T03:39:39.756062+00:00 heroku[web.1]: Process exited with status 126
2021-04-25T03:39:39.861642+00:00 heroku[web.1]: State changed from starting to crashed

SDK version Using latest, Relay Proxy 6.1.6

Language version, developer tools N/A

OS/platform Heroku

Additional context If we can get this to work (seems like a standard file permissions issue), it'd be a really easy way to spin up Relay Proxy.

[eli-darkly]

seems like a standard file permissions issue

Is there some information in particular that makes you think it's a file permissions issue? If you've seen some troubleshooting advice that suggests this, a link would be helpful. The error message sounds more like something one would get from having a binary that was compiled for the wrong architecture.

Also, your steps to reproduce aren't really steps to reproduce, because a required parameter hasn't been filled in: the process type.

[eli-darkly]

Another thing about the steps to reproduce: aren't additional commands after that required to actually start the Heroku container?

[dja]

Hey @eli-darkly, I did some investigation focused on this error: "exec format error", which seems to either be an issue with some file / directory not having the right permissions, a newline character missing from some file (possibly the Dockerfile?), or a problem with an architecture mismatch.

• https://nickjanetakis.com/blog/fixing-exec-format-errors-with-docker-entrypoint-scripts-on-windows
• https://ostechnix.com/fix-exec-format-error-when-running-scripts-with-run-parts-command/
• Last line of this, per architecture choices in build vs run time environments: https://devcenter.heroku.com/articles/build-docker-images-heroku-yml#known-issues-and-limitations

Re: process type - that process type would be something like "web". I'll update the description above, although I'm not sure the naming of it really matters to Heroku.

To your second question - Heroku follows the Dockerfile + ENTRYPOINT to run the app, as you'll see in the first line of the logs where it executes that command. See Heroku's instructions: https://devcenter.heroku.com/articles/container-registry-and-runtime

[eli-darkly]

To your second question - Heroku follows the Dockerfile + ENTRYPOINT to run the app, as you'll see in the first line of the logs where it executes that command. See Heroku's instructions

I didn't mean how does Heroku know what to run in the container, I meant how are you telling Heroku to actually start the container. My reading of the Heroku instructions is that the two commands you showed in your steps to reproduce would only make Heroku aware of the image, and that in order to actually start a service using that image you would need to also use a heroku command, which I didn't see in your steps.

"exec format error", which seems to either be an issue with some file / directory not having the right permissions, a newline character missing from some file (possibly the Dockerfile?), or a problem with an architecture mismatch

Thanks for the links. The first one, about ENTRYPOINT scripts, doesn't seem relevant to this since we don't use a script— as you can see, we're specifying the Relay Proxy executable and its parameters directly: /usr/bin/ldr --config /ldr/ld-relay.conf --allow-missing-file --from-env. If /usr/bin/ldr did not have executable permission then that'd certainly be a problem, but it does; otherwise our many customers who use the Relay Proxy via Docker would be out of luck, the container would not be usable at all by anyone. (Also, that page is not talking about "a newline character missing from some file (possibly the Dockerfile?)"— it's talking about invalid Windows newlines being accidentally put into a shell script. The Dockerfile is only used at the time the container is built, not when you deploy it.)

The second link I think is also not relevant, since it's talking about scripts that are missing a shebang line, and again we're not running a script.

Trying to run an executable that was built for a different architecture is certainly something that could cause "Exec format error", but this image is built for amd64 and it looks to me like that is what Heroku wants. So this is still a puzzler to me.

(sorry about the confusing comment from "LaunchDarklyCI" above - that was me temporarily logged in with the wrong account)

[dja]

Thanks for the investigation here @eli-darkly. Heroku support is telling me that it should use CMD instead of ENTRYPOINT to run it.

From Heroku support:

I think the issue here is that you're using ENTRYPOINT in the Dockerfile from the repository that was linked, this is optional and used to execute binaries or to use images without a shell. This is covered in the Dockerfile commands and runtime section of the Container Registry & Runtime (Docker Deploys) docs. Can you try changing this to CMD and see if that solves the issue?

I tried changing to use CMD in place of ENTRYPOINT, and it's the same issue.

[eli-darkly]

I think that is not quite an accurate description of the meaning of ENTRYPOINT versus CMD, and the support person's summary is slightly different from what Heroku's docs are saying. See below for more details, but I would definitely not expect changing ENTRYPOINT to CMD to fix an "Exec format error" if the problem has to do with an unsupported architecture. You could still try if you want— I'll write up steps for doing this in a separate comment.

As described in the docs for Docker itself, ENTRYPOINT is a completely normal way to specify an executable or command to be run by the container. It can specify either a command line, in which case it'll be run in a shell, or an array, in which case it'll execute the program directly without a shell. In our case it's the latter; there's no purpose in using a shell for something like this where the executable is a real executable and not a script, and where no shell-specific stuff like output redirection is required.

CMD is extremely similar; the difference is basically that it's possible to override the CMD using parameters to docker run, whereas ENTRYPOINT forces the container to always run the same command. So, while we can't ourselves provide support on how to use Heroku and I don't want to argue with Heroku support about Heroku itself, with regard to Docker I think it is misleading to say that "ENTRYPOINT is optional and used to execute binaries or to use images without a shell". ENTRYPOINT and CMD can be used for exactly the same things, and they are both equally optional (you can specify either one or the other, or both).

There is one other slight difference in Heroku. The phrasing in the Heroku docs is: "CMD will always be executed by a shell so that config vars are made available to your process; to execute single binaries or use images without a shell please use ENTRYPOINT." To me, what that is saying is that 1. Heroku has slightly changed the standard behavior of Docker so that CMD will always use a shell even if it wouldn't normally need one (that is, even if you used the array syntax rather than the command-line syntax), and therefore 2. if you do not actually need a shell, they would prefer for you to use ENTRYPOINT, so Heroku can avoid the overhead of running an unnecessary process (very tiny overhead, but for a huge hosting provider, that adds up). Since the Relay Proxy container does not need a shell, it seems to me that what we're doing is exactly what Heroku advises to do.

But either way, with or without a shell, the exact same Relay Proxy executable would be executed.

[eli-darkly]

If you do want to try using CMD instead of ENTRYPOINT to see if that makes a difference in the Heroku environment, here's what I would recommend. This involves fully rebuilding the Docker image from the Relay Proxy source code (it is technically possible to modify an existing image to change parameters that came from the Dockerfile without rebuilding the image from scratch, but it's not straightforward and it's not something we've ever tried so I can't provide any advice about that approach).

1. Check out https://github.com/launchdarkly/ld-relay using Git.
2. Edit Dockerfile, and change ENTRYPOINT to CMD on line 53.
3. Run docker build -t MY_NEW_TAG . (where MY_NEW_TAG is whatever you want to call the new Docker image; note that the final period . is necessary)

This builds the image on your machine but does not push it to Dockerhub. You should then be able to push that tag to Heroku just as you were previously pushing launchdarkly/ld-relay.

That's the simplest way to do it, which doesn't require any tools other than Docker and Git to be installed on your machine (the building of the Relay code happens entirely within Docker using a preconfigured environment).

One caveat is that LaunchDarkly's own release process is slightly different— we build for multiple architectures at once using the Goreleaser tool, and then copy one of those into the Docker image— so it's technically possible for there to be some difference in the resulting binary between that and the simpler process. I wouldn't expect there to be any, at least not any that are significant, but it does mean that if this modified binary works for you it'll be hard to say with 100% certainty that it was due to your change, and not to some other difference. To exactly reproduce our release build, you would need to start a Docker container from the image cimg/go:1.15.2, check out the ld-relay source code inside that container, modify Dockerfile.goreleaser there (instead of Dockerfile), and then run make release inside that container.

[eli-darkly]

Sorry, I should also have said that the instructions above are for Linux or MacOS. I don't think this will work using Docker on Windows.

[cwaldren-ld]

@dja are you still encountering this issue?

[github-actions[bot]]

This issue is marked as stale because it has been open for 30 days without activity. Remove the stale label or comment, or this will be closed in 7 days.