signalfx / maestro-ng

Orchestration of Docker-based, multi-host environments
https://signalfx.com
Apache License 2.0
685 stars 83 forks source link

dependeny management with 'requires' #152

Closed strowi closed 8 years ago

strowi commented 8 years ago

Hi,

i seem to be having a problem with the requires dependency management. For example see the following services. I can start each one on their own without the requires-part.


  registry:
    image: registry:2

    instances:
      registry-01:
        ship: coreos01
        restart: always
        env:  
          REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /var/lib/registry

        ports: { reg: 5000 }

        lifecycle:
          running: [{type: tcp, port: reg }]

        volumes:
          /srv/registry/data: /var/lib/registry

  registry-frontend:
    image: "nginx:1.9"
    requires: [ registry ]

    instances:
      registry-frontend-01:
        ship: coreos01
        restart: always

        ports: 
          https: 
            exposed: 443/tcp
            external: 443/tcp

        lifecycle:
          running: [{type: tcp, port: https}]

        # links:
        #   registry-01: registry
        volumes:
          /srv/registry/auth: /etc/nginx/conf.d
          /srv/registry/certs: /etc/nginx/ssl

But when no service is running and i issue a maestro start registry-frontend it hangs and gives the following error:

% maestro start registry-frontend
  #  INSTANCE                                 SERVICE              SHIP                                     CONTAINER                  STATUS    
  1. registry-frontend-01                     registry-frontend    coreos01 (ssh:37478)                     1.9:2a34274                service did not start! failed to start container! failed!
Traceback (most recent call last):
  File "/home/rvn1/.local/lib/python2.7/site-packages/maestro/__main__.py", line 173, in execute
    getattr(c, options.command)(**vars(options))
  File "/home/rvn1/.local/lib/python2.7/site-packages/maestro/maestro.py", line 299, in start
    auditor=self.auditor).run()
  File "/home/rvn1/.local/lib/python2.7/site-packages/maestro/plays/__init__.py", line 148, in run
    self._end()
  File "/home/rvn1/.local/lib/python2.7/site-packages/maestro/plays/__init__.py", line 140, in _end
    exceptions.raise_with_tb(self._error)
  File "/home/rvn1/.local/lib/python2.7/site-packages/maestro/plays/__init__.py", line 97, in act
    task.run(auditor=self._auditor)
  File "/home/rvn1/.local/lib/python2.7/site-packages/maestro/plays/tasks.py", line 91, in run
    exceptions.raise_with_tb()
  File "/home/rvn1/.local/lib/python2.7/site-packages/maestro/plays/tasks.py", line 85, in run
    self._run()
  File "/home/rvn1/.local/lib/python2.7/site-packages/maestro/plays/tasks.py", line 160, in _run
    '\n'.join(error).strip())
OrchestrationException: Halting start sequence because registry-frontend-01 (on coreos01) failed to start!
2015/10/09 07:50:46 [emerg] 1#1: host not found in upstream "registry:5000" in /etc/nginx/conf.d/registry.conf:2
nginx: [emerg] host not found in upstream "registry:5000" in /etc/nginx/conf.d/registry.conf:2
mpetazzoni commented 8 years ago

By default Maestro doesn't start the dependencies (unless you start everything with maestro start). If you want Maestro to start the dependencies of a given service, you need to specify -d:

maestro start -d registry-frontend

This might feel a little counter intuitive but it turns out to be the "safest" behavior, especially for stopping things. Let me know if that solves your problem!

strowi commented 8 years ago

Hi,

i seem to be running into another issue regarding dependencies...

When i have a service depending on 2 other containers and i restart only one of them, the main-service still looses the connection to the other one:

...
  web2py:
    image: x.y.z/web2py:latest
    requires: [ mysql,redis ]
    instances:
      web2py-01:
        ship: z.local

...

restarting mysql with 'maestro restart -rd mysql` will restart web2py, but loose the connection to the redis-container...

regards, strowi