search

Yooooomi / always-up

Keep your services up by restarting containers based on health checks. Supports dependencies between services.
MIT License
8 stars 0 forks source link

Crash at moderate amount of containers #1

Open amarao opened 1 year ago

amarao commented 1 year ago

I've tested it on a few containers (~ 20) which become unhealthy at random moment: https://github.com/amarao/unhealthy

After about three minutes always-up stops working, and in the next minute crashes with trace:

[DockerOrchestrator]: Starting container /recursing_driscoll
[Manager]: /infallible_faraday is not healthy, restarting
[DockerOrchestrator]: Stopping container /infallible_faraday
[DockerOrchestrator]: Starting container /infallible_faraday
/app/node_modules/docker-modem/lib/modem.js:343
      var msg = new Error(
                ^

Error: (HTTP code 409) unexpected - container is marked for removal and cannot be started 
    at /app/node_modules/docker-modem/lib/modem.js:343:17
    at getCause (/app/node_modules/docker-modem/lib/modem.js:373:7)
    at Modem.buildPayload (/app/node_modules/docker-modem/lib/modem.js:342:5)
    at IncomingMessage.<anonymous> (/app/node_modules/docker-modem/lib/modem.js:310:16)
    at IncomingMessage.emit (node:events:525:35)
    at endReadableNT (node:internal/streams/readable:1359:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  reason: undefined,
  statusCode: 409,
  json: { message: 'container is marked for removal and cannot be started' }
}

Node.js v18.16.1
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

During this moment I didn't performed any Docker operations, so it was pure self-inflicted wound.

Yooooomi commented 1 year ago

Hey! Sad to hear this. I can see a 409 in your logs. Are you using a regular docker installation and using the socket? I haven't tested with 20 containers so that might be the issue. I didn't know your own socket could throw you 409s. I'll have a look this week. Thanks for the report

Yooooomi commented 1 year ago

Ow! I think I know why. Always-up is trying to start a container that has been started with --rm. When it shuts it down, docker tries to remove it and we ask to start it up again. I'll try to see if I can detect that in code directly.