Closed sergiotapia closed 3 months ago
In my server, I can run docker ps
to get a list of running containers. The rails app isn't there.
root@senti:~# docker ps │
│CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES │
│f59d54999b56 traefik:v2.10 "/entrypoint.sh --pr…" 54 minutes ago Up 54 minutes 0.0.0.0:80->80/tcp, :::80->80/tcp traefik │
│b826fe023fe7 postgres:15 "docker-entrypoint.s…" 54 minutes ago Up 54 minutes 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp senti-db
I'm doing a fresh, vanilla Rails app - hoping once I figure this out with help I can provide improvements to the documentation.
Created a brand new rails app, and followed this guide https://gist.github.com/code-206/682c46bd6414b01f912096b4f8865a4b - still see the exact same issue. Don't know where to start to debug this.
Hi @sergiotapia - does docker ps -a
show the container? Is there anything use when you run docker logs senti-web-1f8b5a9fd214d92956810d0072d21919dd2cba40
?
I'm seeing the same problem. relatively new rails app. yesterday I can deploy it onto my staging server. but today it failed on my production server.
Same code. All ubuntus. no logs at all
update 1: I have one cent: maybe the public IP address is blocked by Azure network security group. That leads to the stuck status of my docker container. investigating.
│ INFO [a65dad48] Finished in 0.154 seconds with exit status 0 (successful). │
│Acquiring the deploy lock... │
│Ensure Traefik is running... │
│ INFO [7e25ff8e] Running docker login -u [REDACTED] -p [REDACTED] on 5.161.246.176 │
│ INFO [7e25ff8e] Finished in 0.228 seconds with exit status 0 (successful). │
│ INFO [b2a4abd7] Running docker container start traefik || docker run --name traefik --detach --restart unless-stopped --publish 80:80 --volume /var/run/docker.sock:/var/run/docker.sock --env-file .kamal/env/traefik/traefik.env --log│
│-opt max-size="10m" --label traefik.http.routers.catchall.entryPoints="http" --label traefik.http.routers.catchall.rule="PathPrefix(\`/\`)" --label traefik.http.routers.catchall.service="unavailable" --label traefik.http.routers.catch│
│all.priority="1" --label traefik.http.services.unavailable.loadbalancer.server.port="0" traefik:v2.10 --providers.docker --log.level="DEBUG" on 5.161.246.176 │
│Releasing the deploy lock... │
│ Finished all in 13.8 seconds │
│ ERROR (SSHKit::Command::Failed): Exception while executing on host 5.161.246.176: docker exit status: 125 │
│docker stdout: 9ff6ad229d2e915ecae20efdf2c27d1bc2537d32134df1267c29133d179980e9 │
│docker stderr: Error response from daemon: No such container: traefik │
│Error: failed to start containers: traefik │
│docker: Error response from daemon: driver failed programming external connectivity on endpoint traefik (3c95a0d9f01e0358de9ed494e1d5900c746e05dcaa5f5e2255f5e8b02460f077): Error starting userland proxy: listen tcp4 0.0.0.0:80: bind: a│
│ddress already in use. │
│
Seeing a totally different error today on the same box I left to help troubleshoot the original bug. When I run kamal setup
I see the error above.
I hope v2 of Kamal is better - this is very difficult and if this were a production server/app I would be cooked.
Hi @sergiotapia - does
docker ps -a
show the container? Is there anything use when you rundocker logs senti-web-1f8b5a9fd214d92956810d0072d21919dd2cba40
?
I destroy all the docker containers from the server, then run kamal setup
and see a totally different error than yesterday.
@sergiotapia do you have something like a firewall of your cloud provider or ufw
?
I disabled them and it works again.
Looks like the app container would get stuck if the 80 port is not accessible
Error: failed to start containers: traefik
docker: Error response from daemon: driver failed programming external connectivity on endpoint traefik (3c95a0d9f01e0358de9ed494e1d5900c746e05dcaa5f5e2255f5e8b02460f077): Error starting userland proxy: listen tcp4 0.0.0.0:80: bind: address already in use.
Something is using port 80 on the machine you are trying to deploy to, so Traefik cannot start.
Hi, I'm experiencing something similar. I have an AWS setup with 1 bastion server, 1 web server and 1 worker. Web and worker are in the same private network. port 80 is open
My deployment goes all the way to docker run step, then it start waiting for the health checks.
I got this message
Exception while executing on host <privaye IP>: container not ready (unhealthy)
Then in the server I see the stopped containers
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9f8b7fff4d26 <url>/solorails-kamal-staging:sha "entrypoint.sh" 3 minutes ago Exited (0) 2 minutes ago solorails-web-sha
c345224a88b6 d07ae8c301ce "entrypoint.sh" 10 minutes ago Exited (0) 9 minutes ago solorails-web-44b5b61987472b11bdf14b7da6dee7d5d83d5151_replaced_61750ab791199297
f95411ddfde3 traefik:v2.10 "/entrypoint.sh --pr…" 10 minutes ago Up 10 minutes 0.0.0.0:80->80/tcp, :::80->80/tcp traefik
If try to see the docker logs on the container I get nothing.
The only log I can get is with journalctl but it doesn't make much sense for me.
$ sudo journalctl -u docker
Jun 12 04:19:52 ip-10-0-25-133 systemd[1]: Starting docker.service - Docker Application Container Engine...
Jun 12 04:19:52 ip-10-0-25-133 dockerd[2431]: time="2024-06-12T04:19:52.548928965Z" level=info msg="Starting up"
Jun 12 04:19:52 ip-10-0-25-133 dockerd[2431]: time="2024-06-12T04:19:52.553862826Z" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/syst>
Jun 12 04:19:53 ip-10-0-25-133 dockerd[2431]: time="2024-06-12T04:19:53.115925279Z" level=info msg="Loading containers: start."
Jun 12 04:19:53 ip-10-0-25-133 dockerd[2431]: time="2024-06-12T04:19:53.455704575Z" level=info msg="Loading containers: done."
Jun 12 04:19:53 ip-10-0-25-133 dockerd[2431]: time="2024-06-12T04:19:53.503148846Z" level=info msg="Docker daemon" commit=de5c9cf containerd-snapshotter=false storage-driver=overlay2 version>
Jun 12 04:19:53 ip-10-0-25-133 dockerd[2431]: time="2024-06-12T04:19:53.503364922Z" level=info msg="Daemon has completed initialization"
Jun 12 04:19:53 ip-10-0-25-133 dockerd[2431]: time="2024-06-12T04:19:53.577242755Z" level=info msg="API listen on /run/docker.sock"
Jun 12 04:19:53 ip-10-0-25-133 systemd[1]: Started docker.service - Docker Application Container Engine.
Jun 12 04:28:02 ip-10-0-25-133 dockerd[2431]: time="2024-06-12T04:28:02.214096570Z" level=warning msg="Error while looking up for image " spanID=ec212b71fee47fa4 traceID=9b1f6321dba633e6dfe1>
Jun 12 04:28:05 ip-10-0-25-133 dockerd[2431]: time="2024-06-12T04:28:05.174741560Z" level=warning msg="Error while looking up for image " spanID=da0be03a9008bb96 traceID=3a1ec7a9c52dd086bdcd>
Jun 12 04:28:06 ip-10-0-25-133 dockerd[2431]: time="2024-06-12T04:28:06.412805386Z" level=info msg="ignoring event" container=c345224a88b6452b00783a962018fd75fd6ef319aea15881ed5b42b2b534f13a>
Jun 12 04:28:06 ip-10-0-25-133 dockerd[2431]: time="2024-06-12T04:28:06.819880363Z" level=info msg="ignoring event" container=c345224a88b6452b00783a962018fd75fd6ef319aea15881ed5b42b2b534f13a>
I found my issue, my dockerfile and entrypoint.sh aren't correctly setup.
@hecbuma how did you end up setting up entrypoint.sh? I'm setting up kamal on an existing repo, and everything seems to be running/setting up okay until the entrypoint line and errors out with this:
docker stderr: docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/rails/bin/docker-entrypoint": stat /rails/bin/docker-entrypoint: no such file or directory: unknown.
It seems /rails/bin/docker-entrypoint
file should exist in the vps. Not sure how/when does is it normally created on a new Rails project.
@gauravmc this is my entrypoint.sh
#!/bin/bash
set -e
case $1 in
web)
echo "About to run rake db:migrate and assets:precompile with RAILS_ENV=${RAILS_ENV} in folder ${PWD}"
bundle exec rake db:migrate
bundle exec rake assets:precompile
echo "Additional scripts"
bundle exec rake setup:seed_excluded_clinics
echo "About remove server.pid file and run rails server in folder ${PWD}"
rm -f tmp/pids/server.pid
echo "About to run rails server"
bin/rails server -p 3000 -b '0.0.0.0'
;;
worker)
echo "About to run rake db:migrate and assets:precompile with RAILS_ENV=${RAILS_ENV}"
bundle exec rake db:migrate
echo "About to run rails server"
rm -f tmp/pids/server.pid
bin/rails server -p 3000 -b '0.0.0.0' &
echo "About to run solid_queue:start"
bin/rails solid_queue:start
;;
*)
exec "$@"
;;
esac
exec "$@"
And my dockerfile
FROM --platform=linux/amd64 ruby:3.3.4
RUN apt-get update -qq && apt-get install -y \
build-essential \
nodejs \
postgresql-client
RUN mkdir -p /opt/app
WORKDIR /opt/app
COPY Gemfile Gemfile.lock ./
RUN bundle install
COPY . ./
COPY entrypoint.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/entrypoint.sh
EXPOSE 3000
ENTRYPOINT ["entrypoint.sh"]
@hecbuma Thanks. I end up copy pasting bin/entrypoint from a new Rails 7.1 app, which resolved the issue.
Hi, exploring using Kamal and moving away from PaaS and stuck on this last bit.
My deploy.yml:
And here's my .env
I can successfully do a
kamal env push
. That finishes with no errors.The problem is after doing a
kamal deploy
, the rails app container doesn't get healthy so it fails to deploy.The app container doesn't start, can't see logs to see what's wrong.
Logs: