search

zodern / meteor-up

Production Quality Meteor Deployment to Anywhere
http://meteor-up.com/
MIT License
1.27k stars 280 forks source link

Deployment gets stuck while verifying and breaks afterwards #1357

Closed derwaldgeist closed 4 months ago

derwaldgeist commented 4 months ago

Hey, I redeployed one of my systems recently and noticed that the "Verification" step took ages. Eventually, it breaks most of the time with the error log below (it seems to have tried to start the container 10 times). I already increased the timeout to 240s, but it still happens (I was able to deploy successfully one time, but mostly it breaks). The same happens if I just do mup start. It says "Starting ... Success", but then it breaks while verifying.

I even setup a whole new EC2 instance, but this did not help.

Is there any thing that I can do to troubleshoot this?

I'm using meteor 2.6.1 with mup 1.5.10 in combination with the 'zodern/meteor:root' docker image and mup redis (3.2.10-alpine). Deployment worked fine before, with the same configuration.

------------------------------------STDERR------------------------------------
          17.0.5 port 3000 after 0 ms: Couldn't connect to server
    curl: (7) Failed to connect to 172.17.0.5 port 3000 after 0 ms: Couldn't connect to server
    curl: (7) Failed to connect to 172.17.0.5 port 3000 after 0 ms: Couldn't connect to server
    curl: (7) Failed to connect to 172.17.0.5 port 3000 after 0 ms: Couldn't connect to server
    curl: (7) Failed to connect to 172.17.0.5 port 3000 after 0 ms: Couldn't connect to server
    curl: (7) Failed to connect to 172.17.0.5 port 3000 after 0 ms: Couldn't connect to server
    curl: (7) Failed to connect to 172.17.0.5 port 3000 after 0 ms: Couldn't connect to server
    => Logs:
    => Setting node version
    NODE_VERSION=14.18.3
    Node: 14.18.3 (parsed: 14.18.3)
    Using NVM
    v14.18.3 is already installed.
    Now using node v14.18.3 (npm v6.14.15)
    default -> 14.18.3 (-> v14.18.3 *)
    => Starting meteor app on port 3000
    Monti APM: Successfully connected
    Google Transit: Updating Metro Rail LA
    Google Transit: Updating Metro Bus LA
    Adding cron job "Google Transit stops update"
    Adding cron job "Oplog keep-alive"
    /home/app/scripts/start.sh: line 34:   502 Killed                  node main.js
    => Setting node version
    NODE_VERSION=14.18.3
    Node: 14.18.3 (parsed: 14.18.3)
    Using NVM
    v14.18.3 is already installed.
    Now using node v14.18.3 (npm v6.14.15)
    default -> 14.18.3 (-> v14.18.3 *)
    => Starting meteor app on port 3000
    Monti APM: Successfully connected
    Google Transit: Updating Metro Rail LA
    Google Transit: Updating Metro Bus LA
    Adding cron job "Google Transit stops update"
    Adding cron job "Oplog keep-alive"
    /home/app/scripts/start.sh: line 34:   501 Killed                  node main.js
    => Setting node version
    NODE_VERSION=14.18.3
    Node: 14.18.3 (parsed: 14.18.3)
    Using NVM
    v14.18.3 is already installed.
    Now using node v14.18.3 (npm v6.14.15)
    default -> 14.18.3 (-> v14.18.3 *)
    => Starting meteor app on port 3000
    Monti APM: Successfully connected
    Google Transit: Updating Metro Rail LA
    Google Transit: Updating Metro Bus LA
    Adding cron job "Google Transit stops update"
    Adding cron job "Oplog keep-alive"
    Monti APM: completed instrumenting the app
    /home/app/scripts/start.sh: line 34:   501 Killed                  node main.js
    => Setting node version
    NODE_VERSION=14.18.3
    Node: 14.18.3 (parsed: 14.18.3)
    Using NVM
    v14.18.3 is already installed.
    Now using node v14.18.3 (npm v6.14.15)
    default -> 14.18.3 (-> v14.18.3 *)
    => Starting meteor app on port 3000
    Monti APM: Successfully connected
    Google Transit: Updating Metro Rail LA
    Google Transit: Updating Metro Bus LA
    Adding cron job "Google Transit stops update"
    Adding cron job "Oplog keep-alive"
    Google Transit: Processed 429 stops
    /home/app/scripts/start.sh: line 34:   501 Killed                  node main.js
    => Setting node version
    NODE_VERSION=14.18.3
    Node: 14.18.3 (parsed: 14.18.3)
    Using NVM
    v14.18.3 is already installed.
    Now using node v14.18.3 (npm v6.14.15)
    default -> 14.18.3 (-> v14.18.3 *)
    => Starting meteor app on port 3000
    Monti APM: Successfully connected
    Google Transit: Updating Metro Rail LA
    Google Transit: Updating Metro Bus LA
    Adding cron job "Google Transit stops update"
    Adding cron job "Oplog keep-alive"
    /home/app/scripts/start.sh: line 34:   502 Killed                  node main.js
    => Setting node version
    NODE_VERSION=14.18.3
    Node: 14.18.3 (parsed: 14.18.3)
    Using NVM
    v14.18.3 is already installed.
    Now using node v14.18.3 (npm v6.14.15)
    default -> 14.18.3 (-> v14.18.3 *)
    => Starting meteor app on port 3000
    Monti APM: Successfully connected
    Google Transit: Updating Metro Rail LA
    Google Transit: Updating Metro Bus LA
    Adding cron job "Google Transit stops update"
    Adding cron job "Oplog keep-alive"
    Monti APM: completed instrumenting the app
    /home/app/scripts/start.sh: line 34:   501 Killed                  node main.js
    => Setting node version
    NODE_VERSION=14.18.3
    Node: 14.18.3 (parsed: 14.18.3)
    Using NVM
    v14.18.3 is already installed.
    Now using node v14.18.3 (npm v6.14.15)
    default -> 14.18.3 (-> v14.18.3 *)
    => Starting meteor app on port 3000
    Monti APM: Successfully connected
    Google Transit: Updating Metro Rail LA
    Google Transit: Updating Metro Bus LA
    Adding cron job "Google Transit stops update"
    Adding cron job "Oplog keep-alive"

    => Redeploying previous version of the app

          ------------------------------------STDOUT------------------------------------
          P: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    CONTAINER IP EMPTY
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    CONTAINER IP EMPTY
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    CONTAINER IP EMPTY
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    CONTAINER IP EMPTY
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    CONTAINER IP EMPTY
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    CONTAINER IP EMPTY
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    IP: 172.17.0.5
    => Container status:
    restarted: 10 times {"Bridge":"","SandboxID":"1dfbab8f13b121f5d71de8b2d8df82c3de0016ee9844ae1f6fa002415169095f","SandboxKey":"/var/run/docker/netns/1dfbab8f13b1","Ports":{"3000/tcp":null},"HairpinMode":false,"LinkLocalIPv6Address":"","LinkLocalIPv6PrefixLen":0,"SecondaryIPAddresses":null,"SecondaryIPv6Addresses":null,"EndpointID":"086e4ec1d16f34fa4cb7b66d479f7b6941c2c023306cd2493840b537410f8f6f","Gateway":"172.17.0.1","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"IPAddress":"172.17.0.5","IPPrefixLen":16,"IPv6Gateway":"","MacAddress":"02:42:ac:11:00:05","Networks":{"bridge":{"IPAMConfig":null,"Links":null,"Aliases":null,"MacAddress":"02:42:ac:11:00:05","DriverOpts":null,"NetworkID":"f1ecbcaafaaa8f5ab1775a4da3aa4f7a52214f26a559064ca07485f8d6e11402","EndpointID":"086e4ec1d16f34fa4cb7b66d479f7b6941c2c023306cd2493840b537410f8f6f","Gateway":"172.17.0.1","IPAddress":"172.17.0.5","IPPrefixLen":16,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"DNSNames":null}}} {"Status":"running","Running":true,"Paused":false,"Restarting":false,"OOMKilled":false,"Dead":false,"Pid":51864,"ExitCode":0,"Error":"","StartedAt":"2024-06-28T12:20:58.605439255Z","FinishedAt":"2024-06-28T12:20:58.402000289Z"}

    To see more logs type 'mup logs --tail=200'

If I run mup logs, I get this output:

[staging.marblever.se]v14.18.3 is already installed.
[staging.marblever.se]Now using node v14.18.3 (npm v6.14.15)
[staging.marblever.se]default -> 14.18.3 (-> v14.18.3 *)
[staging.marblever.se]=> Starting meteor app on port 3000
[staging.marblever.se]Monti APM: Successfully connected
[staging.marblever.se]Google Transit: Updating Metro Rail LA
[staging.marblever.se]Google Transit: Updating Metro Bus LA
[staging.marblever.se]Adding cron job "Google Transit stops update"
[staging.marblever.se]Adding cron job "Oplog keep-alive"
[staging.marblever.se]/home/app/scripts/start.sh: line 34:   501 Killed                  node main.js
[staging.marblever.se]=> Setting node version
[staging.marblever.se]NODE_VERSION=14.18.3
[staging.marblever.se]Node: 14.18.3 (parsed: 14.18.3)
[staging.marblever.se]Using NVM
[staging.marblever.se]v14.18.3 is already installed.
[staging.marblever.se]Now using node v14.18.3 (npm v6.14.15)
[staging.marblever.se]default -> 14.18.3 (-> v14.18.3 *)
[staging.marblever.se]=> Starting meteor app on port 3000
[staging.marblever.se]Monti APM: Successfully connected
[staging.marblever.se]Google Transit: Updating Metro Rail LA
[staging.marblever.se]Google Transit: Updating Metro Bus LA
[staging.marblever.se]Adding cron job "Google Transit stops update"
[staging.marblever.se]Adding cron job "Oplog keep-alive"
[staging.marblever.se]/home/app/scripts/start.sh: line 34:   502 Killed                  node main.js
[staging.marblever.se]=> Setting node version
[staging.marblever.se]NODE_VERSION=14.18.3
[staging.marblever.se]Node: 14.18.3 (parsed: 14.18.3)
[staging.marblever.se]Using NVM
[staging.marblever.se]v14.18.3 is already installed.
[staging.marblever.se]Now using node v14.18.3 (npm v6.14.15)
[staging.marblever.se]default -> 14.18.3 (-> v14.18.3 *)
[staging.marblever.se]=> Starting meteor app on port 3000
[staging.marblever.se]Monti APM: Successfully connected
[staging.marblever.se]Google Transit: Updating Metro Rail LA
[staging.marblever.se]Google Transit: Updating Metro Bus LA
[staging.marblever.se]Adding cron job "Google Transit stops update"
[staging.marblever.se]Adding cron job "Oplog keep-alive"
[staging.marblever.se]Monti APM: completed instrumenting the app
[staging.marblever.se]/home/app/scripts/start.sh: line 34:   502 Killed                  node main.js
[staging.marblever.se]=> Setting node version
[staging.marblever.se]NODE_VERSION=14.18.3
[staging.marblever.se]Node: 14.18.3 (parsed: 14.18.3)
[staging.marblever.se]Using NVM
[staging.marblever.se]v14.18.3 is already installed.
[staging.marblever.se]Now using node v14.18.3 (npm v6.14.15)
[staging.marblever.se]default -> 14.18.3 (-> v14.18.3 *)
[staging.marblever.se]=> Starting meteor app on port 3000
[staging.marblever.se]Monti APM: Successfully connected
[staging.marblever.se]Google Transit: Updating Metro Rail LA
[staging.marblever.se]Google Transit: Updating Metro Bus LA
[staging.marblever.se]Adding cron job "Google Transit stops update"
[staging.marblever.se]Adding cron job "Oplog keep-alive"
[staging.marblever.se]/home/app/scripts/start.sh: line 34:   502 Killed                  node main.js
[staging.marblever.se]=> Setting node version
[staging.marblever.se]NODE_VERSION=14.18.3
[staging.marblever.se]Node: 14.18.3 (parsed: 14.18.3)
[staging.marblever.se]Using NVM
[staging.marblever.se]v14.18.3 is already installed.
[staging.marblever.se]Now using node v14.18.3 (npm v6.14.15)
[staging.marblever.se]default -> 14.18.3 (-> v14.18.3 *)
[staging.marblever.se]=> Starting meteor app on port 3000
[staging.marblever.se]Monti APM: Successfully connected
[staging.marblever.se]Google Transit: Updating Metro Rail LA
[staging.marblever.se]Google Transit: Updating Metro Bus LA
[staging.marblever.se]Adding cron job "Google Transit stops update"
[staging.marblever.se]Adding cron job "Oplog keep-alive"
[staging.marblever.se]Google Transit: Processed 429 stops
[staging.marblever.se]/home/app/scripts/start.sh: line 34:   502 Killed                  node main.js
[staging.marblever.se]=> Setting node version
[staging.marblever.se]NODE_VERSION=14.18.3
[staging.marblever.se]Node: 14.18.3 (parsed: 14.18.3)
[staging.marblever.se]Using NVM
[staging.marblever.se]v14.18.3 is already installed.
[staging.marblever.se]Now using node v14.18.3 (npm v6.14.15)
[staging.marblever.se]default -> 14.18.3 (-> v14.18.3 *)
[staging.marblever.se]=> Starting meteor app on port 3000
[staging.marblever.se]Monti APM: Successfully connected
[staging.marblever.se]Google Transit: Updating Metro Rail LA
[staging.marblever.se]Google Transit: Updating Metro Bus LA
[staging.marblever.se]Adding cron job "Google Transit stops update"
[staging.marblever.se]Adding cron job "Oplog keep-alive"
[staging.marblever.se]Monti APM: completed instrumenting the app
[staging.marblever.se]/home/app/scripts/start.sh: line 34:   502 Killed                  node main.js
[staging.marblever.se]=> Setting node version
[staging.marblever.se]NODE_VERSION=14.18.3
[staging.marblever.se]Node: 14.18.3 (parsed: 14.18.3)
[staging.marblever.se]Using NVM
[staging.marblever.se]v14.18.3 is already installed.
[staging.marblever.se]Now using node v14.18.3 (npm v6.14.15)
[staging.marblever.se]default -> 14.18.3 (-> v14.18.3 *)
[staging.marblever.se]=> Starting meteor app on port 3000
[staging.marblever.se]Monti APM: Successfully connected
[staging.marblever.se]Google Transit: Updating Metro Rail LA
[staging.marblever.se]Google Transit: Updating Metro Bus LA
[staging.marblever.se]Adding cron job "Google Transit stops update"
[staging.marblever.se]Adding cron job "Oplog keep-alive"
[staging.marblever.se]/home/app/scripts/start.sh: line 34:   502 Killed                  node main.js
[staging.marblever.se]=> Setting node version
[staging.marblever.se]NODE_VERSION=14.18.3
[staging.marblever.se]Node: 14.18.3 (parsed: 14.18.3)
[staging.marblever.se]Using NVM
[staging.marblever.se]v14.18.3 is already installed.
[staging.marblever.se]Now using node v14.18.3 (npm v6.14.15)
[staging.marblever.se]default -> 14.18.3 (-> v14.18.3 *)
[staging.marblever.se]=> Starting meteor app on port 3000
[staging.marblever.se]Monti APM: Successfully connected
[staging.marblever.se]Google Transit: Updating Metro Rail LA
[staging.marblever.se]Google Transit: Updating Metro Bus LA
[staging.marblever.se]Adding cron job "Google Transit stops update"
[staging.marblever.se]Adding cron job "Oplog keep-alive"
[staging.marblever.se]/home/app/scripts/start.sh: line 34:   502 Killed                  node main.js
[staging.marblever.se]=> Setting node version
[staging.marblever.se]NODE_VERSION=14.18.3
[staging.marblever.se]Node: 14.18.3 (parsed: 14.18.3)
[staging.marblever.se]Using NVM
[staging.marblever.se]v14.18.3 is already installed.
[staging.marblever.se]Now using node v14.18.3 (npm v6.14.15)
[staging.marblever.se]default -> 14.18.3 (-> v14.18.3 *)
[staging.marblever.se]=> Starting meteor app on port 3000
[staging.marblever.se]Monti APM: Successfully connected
[staging.marblever.se]Google Transit: Updating Metro Rail LA
[staging.marblever.se]Google Transit: Updating Metro Bus LA
[staging.marblever.se]Adding cron job "Google Transit stops update"
[staging.marblever.se]Adding cron job "Oplog keep-alive"
[staging.marblever.se]Monti APM: completed instrumenting the app
[staging.marblever.se]/home/app/scripts/start.sh: line 34:   502 Killed                  node main.js
[staging.marblever.se]=> Setting node version
[staging.marblever.se]NODE_VERSION=14.18.3
[staging.marblever.se]Node: 14.18.3 (parsed: 14.18.3)
[staging.marblever.se]Using NVM
[staging.marblever.se]v14.18.3 is already installed.
[staging.marblever.se]Now using node v14.18.3 (npm v6.14.15)
[staging.marblever.se]default -> 14.18.3 (-> v14.18.3 *)
[staging.marblever.se]=> Starting meteor app on port 3000
[staging.marblever.se]Monti APM: Successfully connected
[staging.marblever.se]Google Transit: Updating Metro Rail LA
[staging.marblever.se]Google Transit: Updating Metro Bus LA
[staging.marblever.se]Adding cron job "Google Transit stops update"
[staging.marblever.se]Adding cron job "Oplog keep-alive"
[staging.marblever.se]Google Transit: Processed 429 stops
[staging.marblever.se]/home/app/scripts/start.sh: line 34:   502 Killed                  node main.js
[staging.marblever.se]=> Setting node version
[staging.marblever.se]NODE_VERSION=14.18.3
[staging.marblever.se]Node: 14.18.3 (parsed: 14.18.3)
[staging.marblever.se]Using NVM
[staging.marblever.se]v14.18.3 is already installed.
[staging.marblever.se]Now using node v14.18.3 (npm v6.14.15)
[staging.marblever.se]default -> 14.18.3 (-> v14.18.3 *)
[staging.marblever.se]=> Starting meteor app on port 3000
[staging.marblever.se]Monti APM: Successfully connected
[staging.marblever.se]Google Transit: Updating Metro Rail LA
[staging.marblever.se]Google Transit: Updating Metro Bus LA
[staging.marblever.se]Adding cron job "Google Transit stops update"
[staging.marblever.se]Adding cron job "Oplog keep-alive"
[staging.marblever.se]Google Transit: Processed 429 stops
[staging.marblever.se]/home/app/scripts/start.sh: line 34:   502 Killed                  node main.js
[staging.marblever.se]=> Setting node version
[staging.marblever.se]NODE_VERSION=14.18.3
[staging.marblever.se]Node: 14.18.3 (parsed: 14.18.3)
[staging.marblever.se]Using NVM
[staging.marblever.se]v14.18.3 is already installed.
[staging.marblever.se]Now using node v14.18.3 (npm v6.14.15)
[staging.marblever.se]default -> 14.18.3 (-> v14.18.3 *)
[staging.marblever.se]=> Starting meteor app on port 3000
[staging.marblever.se]Monti APM: Successfully connected
[staging.marblever.se]Google Transit: Updating Metro Rail LA
[staging.marblever.se]Google Transit: Updating Metro Bus LA
[staging.marblever.se]Adding cron job "Google Transit stops update"
[staging.marblever.se]Adding cron job "Oplog keep-alive"
[staging.marblever.se]Monti APM: completed instrumenting the app
[staging.marblever.se]/home/app/scripts/start.sh: line 34:   502 Killed                  node main.js
[staging.marblever.se]=> Setting node version
[staging.marblever.se]NODE_VERSION=14.18.3
[staging.marblever.se]Node: 14.18.3 (parsed: 14.18.3)
[staging.marblever.se]Using NVM
[staging.marblever.se]v14.18.3 is already installed.
[staging.marblever.se]Now using node v14.18.3 (npm v6.14.15)
[staging.marblever.se]default -> 14.18.3 (-> v14.18.3 *)
[staging.marblever.se]=> Starting meteor app on port 3000
[staging.marblever.se]Monti APM: Successfully connected
[staging.marblever.se]Google Transit: Updating Metro Rail LA
[staging.marblever.se]Google Transit: Updating Metro Bus LA
[staging.marblever.se]Adding cron job "Google Transit stops update"
[staging.marblever.se]Adding cron job "Oplog keep-alive"
[staging.marblever.se]Google Transit: Processed 429 stops
[staging.marblever.se]/home/app/scripts/start.sh: line 34:   503 Killed                  node main.js
[staging.marblever.se]=> Setting node version
[staging.marblever.se]NODE_VERSION=14.18.3
[staging.marblever.se]Node: 14.18.3 (parsed: 14.18.3)
[staging.marblever.se]Using NVM
[staging.marblever.se]v14.18.3 is already installed.
[staging.marblever.se]Now using node v14.18.3 (npm v6.14.15)
[staging.marblever.se]default -> 14.18.3 (-> v14.18.3 *)
[staging.marblever.se]=> Starting meteor app on port 3000
[staging.marblever.se]Monti APM: Successfully connected
[staging.marblever.se]Google Transit: Updating Metro Rail LA
[staging.marblever.se]Google Transit: Updating Metro Bus LA
[staging.marblever.se]Adding cron job "Google Transit stops update"
[staging.marblever.se]Adding cron job "Oplog keep-alive"
[staging.marblever.se]/home/app/scripts/start.sh: line 34:   502 Killed                  node main.js
[staging.marblever.se]=> Setting node version
[staging.marblever.se]NODE_VERSION=14.18.3
[staging.marblever.se]Node: 14.18.3 (parsed: 14.18.3)
[staging.marblever.se]Using NVM
[staging.marblever.se]v14.18.3 is already installed.
[staging.marblever.se]Now using node v14.18.3 (npm v6.14.15)
[staging.marblever.se]default -> 14.18.3 (-> v14.18.3 *)
[staging.marblever.se]=> Starting meteor app on port 3000
[staging.marblever.se]Monti APM: Successfully connected
[staging.marblever.se]Google Transit: Updating Metro Rail LA
[staging.marblever.se]Google Transit: Updating Metro Bus LA
[staging.marblever.se]Adding cron job "Google Transit stops update"
[staging.marblever.se]Adding cron job "Oplog keep-alive"
[staging.marblever.se]/home/app/scripts/start.sh: line 34:   502 Killed                  node main.js
[staging.marblever.se]=> Setting node version
[staging.marblever.se]NODE_VERSION=14.18.3
[staging.marblever.se]Node: 14.18.3 (parsed: 14.18.3)
[staging.marblever.se]Using NVM
[staging.marblever.se]v14.18.3 is already installed.
[staging.marblever.se]Now using node v14.18.3 (npm v6.14.15)
[staging.marblever.se]default -> 14.18.3 (-> v14.18.3 *)
[staging.marblever.se]=> Starting meteor app on port 3000
[staging.marblever.se]Monti APM: Successfully connected
[staging.marblever.se]Google Transit: Updating Metro Rail LA
[staging.marblever.se]Google Transit: Updating Metro Bus LA
[staging.marblever.se]Adding cron job "Google Transit stops update"
[staging.marblever.se]Adding cron job "Oplog keep-alive"
[staging.marblever.se]/home/app/scripts/start.sh: line 34:   501 Killed                  node main.js
[staging.marblever.se]=> Setting node version
[staging.marblever.se]NODE_VERSION=14.18.3
[staging.marblever.se]Node: 14.18.3 (parsed: 14.18.3)
[staging.marblever.se]Using NVM
[staging.marblever.se]v14.18.3 is already installed.
[staging.marblever.se]Now using node v14.18.3 (npm v6.14.15)
[staging.marblever.se]default -> 14.18.3 (-> v14.18.3 *)
[staging.marblever.se]=> Starting meteor app on port 3000
[staging.marblever.se]Monti APM: Successfully connected
[staging.marblever.se]Google Transit: Updating Metro Rail LA
[staging.marblever.se]Google Transit: Updating Metro Bus LA
[staging.marblever.se]Adding cron job "Google Transit stops update"
[staging.marblever.se]Adding cron job "Oplog keep-alive"
[staging.marblever.se]Google Transit: Processed 429 stops
[staging.marblever.se]Monti APM: completed instrumenting the app
[staging.marblever.se]/home/app/scripts/start.sh: line 34:   502 Killed                  node main.js
[staging.marblever.se]=> Setting node version
[staging.marblever.se]NODE_VERSION=14.18.3
[staging.marblever.se]Node: 14.18.3 (parsed: 14.18.3)
[staging.marblever.se]Using NVM
[staging.marblever.se]v14.18.3 is already installed.
[staging.marblever.se]Now using node v14.18.3 (npm v6.14.15)
[staging.marblever.se]default -> 14.18.3 (-> v14.18.3 *)
[staging.marblever.se]=> Starting meteor app on port 3000
[staging.marblever.se]Monti APM: Successfully connected
[staging.marblever.se]Google Transit: Updating Metro Rail LA
[staging.marblever.se]Google Transit: Updating Metro Bus LA
[staging.marblever.se]Adding cron job "Google Transit stops update"
[staging.marblever.se]Adding cron job "Oplog keep-alive"
[staging.marblever.se]/home/app/scripts/start.sh: line 34:   502 Killed                  node main.js
[staging.marblever.se]=> Setting node version
[staging.marblever.se]NODE_VERSION=14.18.3
[staging.marblever.se]Node: 14.18.3 (parsed: 14.18.3)
[staging.marblever.se]Using NVM
[staging.marblever.se]v14.18.3 is already installed.
[staging.marblever.se]Now using node v14.18.3 (npm v6.14.15)
[staging.marblever.se]default -> 14.18.3 (-> v14.18.3 *)
[staging.marblever.se]=> Starting meteor app on port 3000
[staging.marblever.se]Monti APM: Successfully connected
[staging.marblever.se]Google Transit: Updating Metro Rail LA
[staging.marblever.se]Google Transit: Updating Metro Bus LA
[staging.marblever.se]Adding cron job "Google Transit stops update"
[staging.marblever.se]Adding cron job "Oplog keep-alive"
[staging.marblever.se]Google Transit: Processed 429 stops
[staging.marblever.se]/home/app/scripts/start.sh: line 34:   502 Killed                  node main.js
[staging.marblever.se]=> Setting node version
[staging.marblever.se]NODE_VERSION=14.18.3
[staging.marblever.se]Node: 14.18.3 (parsed: 14.18.3)
[staging.marblever.se]Using NVM
[staging.marblever.se]v14.18.3 is already installed.
[staging.marblever.se]Now using node v14.18.3 (npm v6.14.15)
[staging.marblever.se]default -> 14.18.3 (-> v14.18.3 *)
[staging.marblever.se]=> Starting meteor app on port 3000
[staging.marblever.se]Monti APM: Successfully connected
[staging.marblever.se]Google Transit: Updating Metro Rail LA
[staging.marblever.se]Google Transit: Updating Metro Bus LA
[staging.marblever.se]Adding cron job "Google Transit stops update"
[staging.marblever.se]Adding cron job "Oplog keep-alive"
[staging.marblever.se]/home/app/scripts/start.sh: line 34:   502 Killed                  node main.js
[staging.marblever.se]=> Setting node version
[staging.marblever.se]NODE_VERSION=14.18.3
[staging.marblever.se]Node: 14.18.3 (parsed: 14.18.3)
[staging.marblever.se]Using NVM
[staging.marblever.se]v14.18.3 is already installed.
[staging.marblever.se]Now using node v14.18.3 (npm v6.14.15)
[staging.marblever.se]default -> 14.18.3 (-> v14.18.3 *)
[staging.marblever.se]=> Starting meteor app on port 3000

Using mup status, I can see the following red lines indicating an error:

    App available through reverse proxy
      - Available in app's docker container: false
derwaldgeist commented 4 months ago

I analyzed this a bit further and noticed that mup deploys successfully if I disable a job that runs at server start. This job retrieves data from Google Transit and processes it. The thing is: This job only takes about 4s to finish, and the problem occurs even if I put it in an async thread, using Meteor.defer.

How can this affect mup's startup verification, and why didn't this happen before?

zodern commented 4 months ago

Usually the log message [staging.marblever.se]/home/app/scripts/start.sh: line 34: 502 Killed node main.js is from the server running out of memory and linux stopping the app.

derwaldgeist commented 4 months ago

@zodern Thanks for the fast response. That's interesting. The data I am processing is not very big, though. And it takes just 4s to process locally on my dev machine.

Closing this issue now, since it might not be related to mup. Or might there be any reasons why mup itself uses a lot of memory?

It's also strange that the verifying process takes ages, even if it runs through. I did not experience this behaviour before, using the same server configuration. What exactly is mup doing when "verifying"?

zodern commented 4 months ago

Or might there be any reasons why mup itself uses a lot of memory?

Unless you are deploying multiple apps to the server at the same time, I wouldn't expect mup itself to take much memory. One option would be to ssh onto the server during the deploy and run htop or docker stats and monitor how much memory is being used.

What exactly is mup doing when "verifying"?

It uses curl to try loading the app until it succeeds or it runs out of time (the default is 60 seconds). Each time it gets the container's current ip address. Sometimes the container doesn't have an ip address at the moment, usually due to the app crashing and the container in the process of restarting. If that happens enough times, it will fail the deploy early instead of waiting the full amount of time.

https://github.com/zodern/meteor-up/blob/2272e108dc2c1b227398b175e81dc4b8772e1745/src/plugins/meteor/assets/meteor-deploy-check.sh#L53-L114

derwaldgeist commented 4 months ago

Thanks for sharing these insights. I'll investigate further.