search

zodern / meteor-up

Production Quality Meteor Deployment to Anywhere
http://meteor-up.com/
MIT License
1.27k stars 280 forks source link

New server deploy fails Verifying Deployment for known good app #1320

Closed kevpfowler closed 1 year ago

kevpfowler commented 2 years ago

I have a meteor app that I deploy successfully to two servers already (running ancient ubuntu 16.04). I spun up a new ubuntu 20.02 EC2, and am trying to deploy to that. This has been spectacularly unsuccessful. I have tried numerous things but the Verify stage always fails.

If I go on to the docker container where the app is, I can see that it is not listening on port 80 - it is not listening on any port. The server logs look normal up until it runs node main.js. At that point none of our usual log messages are seen. But no errors are seen either, and the node process is running.

We have confidence in the app code because we deploy the same version on two other servers successfully. I had started with ubuntu 22.04, but on seeing some other issues here I went down to 20.04, but that does not seem to be the problem. (Also tried with 18.04)

This server sits behind a NAT server, which is why the ssh port is 10022 (it is port-mapped to this server's port 22, and that works fine). We also have nginx running on the NAT server, forwarding requests to our domain name to this server's port 80. This precisely mimics another setup we have that is successfully deployed/running.

Any ideas what I'm missing?

Mup version (mup --version): 1.5.9

Mup config

{
  "servers": {
    "one": {
      "host": "subdomain.subdomain.host.com",
      "username": "ubuntu",
      "opts": {
        "port": 10022
      }
    }
  },
  "proxy": {
    "domains": "subdomain.subdomain.host.com"
  },
  "app": {
    "name": "my-app",
    "path": "/home/kfowler/Gitlab/appName",
    "docker": {
      "image": "zodern/meteor:root",
      "imagePort": 80,
      "bind": "127.0.0.1",
      "prepareBundle": true,
      "stopAppDuringPrepareBundle": false,
      "useBuildKit": true
    },
    "servers": {
      "one": {}
    },
    "buildOptions": {
      "serverOnly": true,
      "debug": true,
      "cleanAfterBuild": true,
      "allowIncompatibleUpdates": true,
      "buildLocation": "/tmp/mup-meteor-0a4743e6-0d43-45c7-b50b-28b493b8499a",
      "cleanBuildLocation": true
    },
    "env": {
      "ROOT_URL": "https://subdomain.subdomain.host.com",
      "MONGO_URL": "mongodb+srv://subdomain.subdomain.host.com/meteor?authSource=%24external&authMechanism=MONGODB-AWS&retryWrites=true&w=majority",
      "VIRTUAL_HOST": "subdomain.subdomain.host.com",
      "HTTPS_METHOD": "noredirect",
      "VIRTUAL_PORT": 80,
      "HTTP_FORWARDED_COUNT": 1
    },
    "log": {
      "driver": "json-file"
    },
    "deployCheckWaitTime": 60,
    "deployCheckPort": 80,
    "enableUploadProgressBar": true,
    "type": "meteor"
  }
}
✓ Config is valid

Output of command (mup deploy)

ot building app. Using build from previous deploy at
/tmp/mup-meteor-0a4743e6-0d43-45c7-b50b-28b493b8499a

Started TaskList: Pushing Meteor App
[subdomain.subdomain.host.com] - Pushing Meteor App Bundle to the Server
[subdomain.subdomain.host.com] - Pushing Meteor App Bundle to the Server: SUCCESS

Started TaskList: Prepare App Bundle
[subdomain.subdomain.host.com] - Prepare Bundle
Updating base image
root: Pulling from zodern/meteor
Digest: sha256:0a2d5104d177d06131b39d7321e2edd8ecd301d438c188e55d9ac173c0ab10fb
Status: Image is up to date for zodern/meteor:root
docker.io/zodern/meteor:root
Preparing for docker build
Creating Dockerfile
Finished creating Dockerfile
Building image
#1 [internal] load build definition from Dockerfile
#1 sha256:8f14b59580a86feed32cf72ab389cc97d18b26b16de64e5e1fd76cd49b3f65f2
#1 transferring dockerfile: 682B 0.0s done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:cf6ff717b55ac4f6695a5e6aa1748ba932dd822f1416dd75396912f0d22e88dd
#2 transferring context: 2B done
#2 DONE 0.0s

#3 resolve image config for docker.io/docker/dockerfile:1-experimental
#3 sha256:69b62bd5db81d5b3c4a4dffa5235d4a66365d82e081ec371170be889e64c7032
#3 DONE 0.3s

#4 docker-image://docker.io/docker/dockerfile:1-experimental@sha256:600e5c62eedff338b3f7a0850beb7c05866e0ef27b2d2e8c02aa468e78496ff5
#4 sha256:d6fa9fb52f18bc64cd1e721ee71060639a9ab5f6c484cd06bfac8d8a11d6c802
#4 CACHED

#5 [internal] load metadata for docker.io/zodern/meteor:root
#5 sha256:f02a8236c687f1b378aeb0f8f78b4e0fd54c34f9d5cf81976648c2fbbcee9efe
#5 DONE 0.0s

#6 [1/4] FROM docker.io/zodern/meteor:root
#6 sha256:c199e50e044cb8fbe6ebd6d596cb55099c52ebe1c21a2290d9572b4469086675
#6 DONE 0.0s

#9 [internal] load build context
#9 sha256:2cb43a7724d41140d765a61fc7bbab5dd2e1620bc51f608a9076abf2d7ccd2de
#9 transferring context: 78.54MB 0.8s done
#9 DONE 0.8s

#7 [2/4] RUN bash ./scripts/onbuild-node.sh
#7 sha256:c646b4a36b1c05b86bc3a74e15c0e36cf49bdb49ba8a64b5af0b565c794fe087
#7 CACHED

#8 [3/4] RUN mkdir /built_app || true
#8 sha256:a4629c993b3330501771e247dfab4081367608007708e707357753bb396005ef
#8 CACHED

#10 [4/4] RUN --mount=type=bind,target=/tmp/__mup-bundle tar -xzf /tmp/__mup-bundle/bundle.tar.gz -C /built_app --strip-components=1 && ls /built_app
#10 sha256:e1ed77970286b46244862488f7a154819a61141c874d96ae848c2a95221c24e1
#10 CACHED

#11 [5/4] RUN cd /built_app/programs/server &&     npm install --unsafe-perm
#11 sha256:de5f0be48ca0d29af78d7c3d4cf6fcfc6fbb7e11e9315d71282e26dfbf3d58c1
#11 CACHED

#12 exporting to image
#12 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#12 exporting layers done
#12 writing image sha256:68cd116054146fbb8d93f4a10b21b1d0345f007e8f9c2b13bac45e4a259763e3 done
#12 naming to docker.io/library/mup-super:build done
#12 DONE 0.0s

real    0m2.154s
user    0m0.423s
sys     0m0.139s
Tagged latest
Total reclaimed space: 0B
Total reclaimed space: 0B
[subdomain.subdomain.host.com] - Prepare Bundle: SUCCESS

Started TaskList: Configuring App
[subdomain.subdomain.host.com] - Pushing the Startup Script
[subdomain.subdomain.host.com] - Pushing the Startup Script: SUCCESS
[subdomain.subdomain.host.com] - Sending Environment Variables
[subdomain.subdomain.host.com] - Sending Environment Variables: SUCCESS

Started TaskList: Start Meteor
[subdomain.subdomain.host.com] - Start Meteor
using image
removing last
Image mup-appname:latest
Volume
Removing docker containers. Errors about nonexistent endpoints and containers are normal.
super
Error response from daemon: endpoint appname not found
Error: No such container: appname-frontend
Error response from daemon: endpoint appname-frontend not found
Error: No such container: appname-nginx-letsencrypt
Error response from daemon: endpoint appname-nginx-letsencrypt not found
Error: No such container: appname-nginx-proxy
Error response from daemon: endpoint appname-nginx-proxy not found
Finished removing docker containers
31d61088e03a700578985390a805602d5eb77be0d58c21bc6fbe2781a0b59b79
Ran zodern/meteor:root
[subdomain.subdomain.host.com] - Start Meteor: SUCCESS
[subdomain.subdomain.host.com] - Verifying Deployment
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
IP: 172.17.0.4
curl: (7) Failed to connect to 172.17.0.4 port 80: Connection refused
=> Container status:
restarted: 0 times {"Bridge":"","SandboxID":"a23f14fcef1673489fed1072e71b3144a528a7ea91766b286025111003d70707","HairpinMode":false,"LinkLocalIPv6Address":"","LinkLocalIPv6PrefixLen":0,"Ports":{"3000/tcp":null,"80/tcp":null},"SandboxKey":"/var/run/docker/netns/a23f14fcef16","SecondaryIPAddresses":null,"SecondaryIPv6Addresses":null,"EndpointID":"9d73a4cc9c0b89a0bba8bce4302020cb57d0467e1c70454e2e4c8f5b46f0409d","Gateway":"172.17.0.1","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"IPAddress":"172.17.0.4","IPPrefixLen":16,"IPv6Gateway":"","MacAddress":"02:42:ac:11:00:04","Networks":{"bridge":{"IPAMConfig":null,"Links":null,"Aliases":null,"NetworkID":"e84509d3e31ec3e8717f6d0e916d0fbafc526088cb0262f56a08050240efb7a2","EndpointID":"9d73a4cc9c0b89a0bba8bce4302020cb57d0467e1c70454e2e4c8f5b46f0409d","Gateway":"172.17.0.1","IPAddress":"172.17.0.4","IPPrefixLen":16,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"MacAddress":"02:42:ac:11:00:04","DriverOpts":null}}} {"Status":"running","Running":true,"Paused":false,"Restarting":false,"OOMKilled":false,"Dead":false,"Pid":10394,"ExitCode":0,"Error":"","StartedAt":"2022-07-08T13:19:53.827018931Z","FinishedAt":"0001-01-01T00:00:00Z"}
=> Logs:
=> Setting node version
NODE_VERSION=14.18.3
v14.18.3 is already installed.
Now using node v14.18.3 (npm v6.14.15)
default -> 14.18.3 (-> v14.18.3 *)
=> Starting meteor app on port 80

To see more logs type 'mup logs --tail=200'

=> Redeploying previous version of the app

mup status:

=> Servers
  - subdomain.subdomain.host.com: Ubuntu 20.04

=> Docker Status
 - ssubdomain.subdomain.host.com: 20.10.17 Running

=> Meteor Status - super
  - subdomain.subdomain.host.com: running
    Created at 2022-07-08T13:20:58.44121028Z
    Restarted 0 times
    ENV:
      - ROOT_URL=https://subdomain.subdomain.host.com/
      - MONGO_URL=mongodb+srv://subdomain.subdomain.host.com/meteor?authSource=%24external&authMechanism=MONGODB-AWS&retryWrites=true&w=majority
      - VIRTUAL_HOST=subdomain.subdomain.host.com
      - HTTPS_METHOD=noredirect
      - VIRTUAL_PORT=80
      - HTTP_FORWARDED_COUNT=1
      - METEOR_SETTINGS={"version":{"commitId":"fd4210e2426fef54aa85886e93c0a046bc5f5b3e","commitTimestamp":"2022-07-07 08:21:17 -0400","commitDate":"2022-07-07"}}
      - PORT=80
      - PATH=/home/app/.onbuild-node/bin:/home/app/.onbuild-node/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      - NODE_PATH=/home/app/.onbuild-node/lib/node_modules
    Exposed Ports:
      - 3000/tcp
      - 80/tcp
    App available through reverse proxy
      - Available in app's docker container: false

=> Reverse Proxy Status
 - subdomain.subdomain.host.com:
   - NGINX:
     - Status: running
     - Ports:
       - HTTPS: 443
       - HTTP: 80
   - Let's Encrypt
     - Status: running
kevpfowler commented 2 years ago

These are the logs I see in the application contationer:

~$ docker logs appname
=> Setting node version
NODE_VERSION=14.18.3
v14.18.3 is already installed.
Now using node v14.18.3 (npm v6.14.15)
default -> 14.18.3 (-> v14.18.3 *)
=> Starting meteor app on port 80

Normally following that I see the logs from my app, or an error message that would help me to debug. But that is all I get.

sayandcode commented 1 year ago

@kevpfowler Did you manage to solve the issue, or find any workarounds?

kevpfowler commented 1 year ago

I did not. After burning too much time on the issue I stayed with our original setup, and we went a non-meteor direction for our new application anyway. I'm closing this as it seems somehow unique to our particular scheme/config.