search

att-comdev / armada

This project has moved to OpenStack.
https://www.airshipit.org/
Apache License 2.0
78 stars 35 forks source link

Unable to execute 'get_status' API using Armada Client #180

Open eanylin opened 6 years ago

eanylin commented 6 years ago

Is this a bug report or feature request? (choose one): This is a bug

Python Version (output of python --version): Python 3.5.2

Development or Deployment Environment?: Deployment Environment (CI/CD lab)

Release Tag or Master: Master

Expected Behavior: Armada should be able check that Tiller is up and running properly with get_status API call

What Actually Happened: Problem seems to be related to armada config. We see the following errors in the logs of the Armada API container

Operational MODE: single process Loading paste environment: config:/etc/armada/api-paste.ini Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp return loadobj(APP, uri, name=name, kw) File "/usr/local/lib/python3.5/dist-packages/paste/deploy/loadwsgi.py", line 271, in loadobj global_conf=global_conf) File "/usr/local/lib/python3.5/dist-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext global_conf=global_conf) File "/usr/local/lib/python3.5/dist-packages/paste/deploy/loadwsgi.py", line 320, in _loadconfig return loader.get_context(object_type, name, global_conf) File "/usr/local/lib/python3.5/dist-packages/paste/deploy/loadwsgi.py", line 450, in get_context global_additions=global_additions) File "/usr/local/lib/python3.5/dist-packages/paste/deploy/loadwsgi.py", line 559, in _pipeline_app_context APP, pipeline[-1], global_conf) File "/usr/local/lib/python3.5/dist-packages/paste/deploy/loadwsgi.py", line 458, in get_context section) File "/usr/local/lib/python3.5/dist-packages/paste/deploy/loadwsgi.py", line 517, in _context_from_explicit value = import_string(found_expr) File "/usr/local/lib/python3.5/dist-packages/paste/deploy/loadwsgi.py", line 22, in import_string return pkg_resources.EntryPoint.parse("x=" + s).load(False) File "/usr/local/lib/python3.5/dist-packages/pkg_resources/init.py", line 2405, in load return self.resolve() File "/usr/local/lib/python3.5/dist-packages/pkg_resources/init.py", line 2411, in resolve module = import(self.module_name, fromlist=['name'], level=0) File "./armada/api/server.py", line 31, in conf.set_app_default_configs() File "./armada/conf/init.py", line 31, in set_app_default_configs set_default_for_default_log_levels() File "./armada/conf/init.py", line 47, in set_default_for_default_log_levels log.register_options(CONF) File "/usr/local/lib/python3.5/dist-packages/oslo_log/log.py", line 250, in register_options conf.register_cli_opts(_options.common_cli_opts) File "/usr/local/lib/python3.5/dist-packages/oslo_config/cfg.py", line 2426, in __inner result = f(self, *args, *kwargs) File "/usr/local/lib/python3.5/dist-packages/oslo_config/cfg.py", line 2648, in register_cli_opts self.register_cli_opt(opt, group, clear_cache=False) File "/usr/local/lib/python3.5/dist-packages/oslo_config/cfg.py", line 2430, in __inner return f(self, args, kwargs) File "/usr/local/lib/python3.5/dist-packages/oslo_config/cfg.py", line 2640, in register_cli_opt raise ArgsAlreadyParsedError("cannot register CLI option") oslo_config.cfg.ArgsAlreadyParsedError: arguments already parsed: cannot register CLI option

How to Reproduce the Issue (as minimally as possible): Re-run the genesis-full pipeline in CI/CD lab to start deployment from scratch

Any Additional Comments: N.A.

gardlt commented 6 years ago

https://review.gerrithub.io/#/c/390066/

eanylin commented 6 years ago

We tested with the patch set in CI/CD lab but were unable to deploy the UCP components with the new test image (http://docker.io/gerrive/armada:git-4470f4e). We are seeing the below errors.

Seems like the new changes caused some side effects with deployment of UCP components.

ubuntu@genesis:~$ sudo kubectl get pods -n ucp -o wide
NAME                         READY     STATUS    RESTARTS   AGE       IP              NODE
etcd-5bcbbd679c-mf4cr        1/1       Running   0          9m        10.97.244.208   genesis
mariadb-0                    1/1       Running   0          9m        10.97.244.210   genesis
mariadb-1                    1/1       Running   0          6m        10.97.244.212   genesis
mariadb-2                    1/1       Running   0          5m        10.97.244.213   genesis
memcached-5bf49657db-rfxsk   1/1       Running   0          9m        10.97.244.207   genesis
postgresql-0                 1/1       Running   0          9m        10.97.244.211   genesis
rabbitmq-f68649644-7w49s     1/1       Running   0          9m        10.97.244.209   genesis
ubuntu@genesis:~$
2017-12-02 02:21:25.091 1 INFO armada.handlers.armada [-] Deploying: UCP Services
2017-12-02 02:21:25.221 1 INFO armada.handlers.chartbuilder [-] Building dependency chart maas for release maas
2017-12-02 02:21:25.259 1 DEBUG armada.handlers.armada [-] RELEASE: maas sync /usr/local/lib/python3.5/dist-packages/armada/handlers/armada.py:253
2017-12-02 02:21:25.259 1 INFO armada.handlers.armada [-] Installing release maas
2017-12-02 02:21:25.265 1 INFO armada.handlers.tiller [-] Wait: False, Timeout: 3600
2017-12-02 02:21:25.319 1 CRITICAL armada [-] Unhandled error: armada.exceptions.tiller_exceptions.GetReleaseStatusException: Failed to get ucp-maas status 0 version
2017-12-02 02:21:25.319 1 ERROR armada Traceback (most recent call last):
2017-12-02 02:21:25.319 1 ERROR armada   File "/usr/local/lib/python3.5/dist-packages/armada/handlers/tiller.py", line 325, in install_release
2017-12-02 02:21:25.319 1 ERROR armada     release_request, self.timeout, metadata=self.metadata)
2017-12-02 02:21:25.319 1 ERROR armada   File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 492, in __call__
2017-12-02 02:21:25.319 1 ERROR armada     return _end_unary_response_blocking(state, call, False, deadline)
2017-12-02 02:21:25.319 1 ERROR armada   File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 440, in _end_unary_response_blocking
2017-12-02 02:21:25.319 1 ERROR armada     raise _Rendezvous(state, None, None, deadline)
2017-12-02 02:21:25.319 1 ERROR armada grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, render error in "maas/deployment-rack.yaml": template: maas/deployment-rack.yaml:35:62: executing "maas/deployment-rack.yaml" at <include "helm-toolki...>: error calling include: template: maas/charts/helm-toolkit/utils/_hash.tpl:22:4: executing "helm-toolkit.utils.hash" at <include $wtf $contex...>: error calling include: template: maas/configmap-bin.yaml:39:52: executing "maas/configmap-bin.yaml" at <include "helm-toolki...>: error calling include: template: maas/charts/helm-toolkit/utils/_template.tpl:22:3: executing "helm-toolkit.utils.template" at <include $wtf $contex...>: error calling include: template: no template "maas/bin/_register-rack-controller.sh.tpl" associated with template "gotpl")>
2017-12-02 02:21:25.319 1 ERROR armada 
2017-12-02 02:21:25.319 1 ERROR armada During handling of the above exception, another exception occurred:
2017-12-02 02:21:25.319 1 ERROR armada 
2017-12-02 02:21:25.319 1 ERROR armada Traceback (most recent call last):
2017-12-02 02:21:25.319 1 ERROR armada   File "/usr/local/lib/python3.5/dist-packages/armada/handlers/tiller.py", line 382, in get_release_status
2017-12-02 02:21:25.319 1 ERROR armada     status_request, self.timeout, metadata=self.metadata)
2017-12-02 02:21:25.319 1 ERROR armada   File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 492, in __call__
2017-12-02 02:21:25.319 1 ERROR armada     return _end_unary_response_blocking(state, call, False, deadline)
2017-12-02 02:21:25.319 1 ERROR armada   File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 440, in _end_unary_response_blocking
2017-12-02 02:21:25.319 1 ERROR armada     raise _Rendezvous(state, None, None, deadline)
2017-12-02 02:21:25.319 1 ERROR armada grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, getting deployed release "ucp-maas": release: "ucp-maas" not found)>
2017-12-02 02:21:25.319 1 ERROR armada 
2017-12-02 02:21:25.319 1 ERROR armada During handling of the above exception, another exception occurred:
2017-12-02 02:21:25.319 1 ERROR armada 
2017-12-02 02:21:25.319 1 ERROR armada Traceback (most recent call last):
2017-12-02 02:21:25.319 1 ERROR armada   File "/usr/local/bin/armada", line 10, in <module>
2017-12-02 02:21:25.319 1 ERROR armada     sys.exit(main())
2017-12-02 02:21:25.319 1 ERROR armada   File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 722, in __call__
2017-12-02 02:21:25.319 1 ERROR armada     return self.main(*args, **kwargs)
2017-12-02 02:21:25.319 1 ERROR armada   File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 697, in main
2017-12-02 02:21:25.319 1 ERROR armada     rv = self.invoke(ctx)
2017-12-02 02:21:25.319 1 ERROR armada   File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 1066, in invoke
2017-12-02 02:21:25.319 1 ERROR armada     return _process_result(sub_ctx.command.invoke(sub_ctx))
2017-12-02 02:21:25.319 1 ERROR armada   File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 895, in invoke
2017-12-02 02:21:25.319 1 ERROR armada     return ctx.invoke(self.callback, **ctx.params)
2017-12-02 02:21:25.319 1 ERROR armada   File "/usr/local/lib/python3.5/dist-packages/click/core.py", line 535, in invoke
2017-12-02 02:21:25.319 1 ERROR armada     return callback(*args, **kwargs)
2017-12-02 02:21:25.319 1 ERROR armada   File "/usr/local/lib/python3.5/dist-packages/click/decorators.py", line 17, in new_func
2017-12-02 02:21:25.319 1 ERROR armada     return f(get_current_context(), *args, **kwargs)
2017-12-02 02:21:25.319 1 ERROR armada   File "/usr/local/lib/python3.5/dist-packages/armada/cli/apply.py", line 120, in apply_create
2017-12-02 02:21:25.319 1 ERROR armada     wait).invoke()
2017-12-02 02:21:25.319 1 ERROR armada   File "/usr/local/lib/python3.5/dist-packages/armada/cli/apply.py", line 186, in invoke
2017-12-02 02:21:25.319 1 ERROR armada     resp = armada.sync()
2017-12-02 02:21:25.319 1 ERROR armada   File "/usr/local/lib/python3.5/dist-packages/armada/handlers/armada.py", line 330, in sync
2017-12-02 02:21:25.319 1 ERROR armada     timeout=chart_timeout)
2017-12-02 02:21:25.319 1 ERROR armada   File "/usr/local/lib/python3.5/dist-packages/armada/handlers/tiller.py", line 328, in install_release
2017-12-02 02:21:25.319 1 ERROR armada     status = self.get_release_status(release)
2017-12-02 02:21:25.319 1 ERROR armada   File "/usr/local/lib/python3.5/dist-packages/armada/handlers/tiller.py", line 385, in get_release_status
2017-12-02 02:21:25.319 1 ERROR armada     raise ex.GetReleaseStatusException(release, version)
2017-12-02 02:21:25.319 1 ERROR armada armada.exceptions.tiller_exceptions.GetReleaseStatusException: Failed to get ucp-maas status 0 version
2017-12-02 02:21:25.319 1 ERROR armada