controller won't start due to secrets issue

[jdumars]

ENV: Kubernetes:

Client Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.0+5cb86ee", GitCommit:"5cb86ee022267586db386f62781338b0483733b3", GitTreeState:"not a git tree"}
Server Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.0", GitCommit:"5cb86ee022267586db386f62781338b0483733b3", GitTreeState:"clean"}`

Running in AWS (separate k8s masters, etcd cluster, nodes), CoreOS 899.11.0, workflow-dev

Pods:

ENG000657:scripts jasondumars$ kubectl get pods --namespace=deis
NAME                          READY     STATUS             RESTARTS   AGE
deis-builder-wm8uo            1/1       Running            1          16m
deis-controller-azbqs         0/1       CrashLoopBackOff   8          13m
deis-database-7qvq2           1/1       Running            0          16m
deis-logger-e1lxe             1/1       Running            0          16m
deis-logger-fluentd-ehrb0     1/1       Running            0          16m
deis-logger-fluentd-r4hx3     1/1       Running            0          16m
deis-minio-mswrt              1/1       Running            0          16m
deis-monitor-grafana-x3yct    1/1       Running            0          16m
deis-monitor-influxdb-mcogd   1/1       Running            0          16m
deis-monitor-stdout-8glu3     1/1       Running            0          16m
deis-monitor-telegraf-133mg   1/1       Running            0          16m
deis-monitor-telegraf-uudiz   1/1       Running            0          16m
deis-registry-qail0           1/1       Running            3          16m
deis-router-1cioi             1/1       Running            0          16m
deis-workflow-manager-iee7k   1/1       Running            0          16m

Controller will not start, and errors out with this:

$ kubectl --namespace=deis logs deis-controller-azbqs -p
system information:
Django Version: 1.9.6
Python 3.5.1
Django checks:
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/django/core/management/base.py", line 348, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3.5/site-packages/django/core/management/base.py", line 391, in execute
    saved_locale = translation.get_language()
  File "/usr/lib/python3.5/site-packages/django/utils/translation/__init__.py", line 183, in get_language
    return _trans.get_language()
  File "/usr/lib/python3.5/site-packages/django/utils/translation/__init__.py", line 57, in __getattr__
    if settings.USE_I18N:
  File "/usr/lib/python3.5/site-packages/django/conf/__init__.py", line 55, in __getattr__
    self._setup(name)
  File "/usr/lib/python3.5/site-packages/django/conf/__init__.py", line 43, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/lib/python3.5/site-packages/django/conf/__init__.py", line 120, in __init__
    raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/manage.py", line 12, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/lib/python3.5/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3.5/site-packages/django/core/management/__init__.py", line 345, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3.5/site-packages/django/core/management/base.py", line 360, in run_from_argv
    connections.close_all()
  File "/usr/lib/python3.5/site-packages/django/db/utils.py", line 230, in close_all
    for alias in self:
  File "/usr/lib/python3.5/site-packages/django/db/utils.py", line 224, in __iter__
    return iter(self.databases)
  File "/usr/lib/python3.5/site-packages/django/utils/functional.py", line 33, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/lib/python3.5/site-packages/django/db/utils.py", line 157, in databases
    self._databases = settings.DATABASES
  File "/usr/lib/python3.5/site-packages/django/conf/__init__.py", line 55, in __getattr__
    self._setup(name)
  File "/usr/lib/python3.5/site-packages/django/conf/__init__.py", line 43, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/lib/python3.5/site-packages/django/conf/__init__.py", line 120, in __init__
    raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty. 

Secret appears ok:

$ kubectl get secrets django-secret-key -o yaml --namespace=deis apiVersion: v1 data: secret-key: XxXxayBgSltIcFMzP14oakpFSVFxZVs/cyV8I3RMRXpFPF1yRkpDNHczPHhefT9tUHJGbkpiQEE2ZXZxS0dVbQ== kind: Secret metadata: annotations: chart.helm.sh/description: For testing only! chart.helm.sh/file: /Users/jasondumars/.helmc/workspace/charts/workflow-dev/manifests/deis-controller-secret-django-secret-key.yaml chart.helm.sh/name: workflow-dev chart.helm.sh/version: v2-beta creationTimestamp: 2016-05-11T17:20:35Z labels: heritage: deis name: django-secret-key namespace: deis resourceVersion: "16640" selfLink: /api/v1/namespaces/deis/secrets/django-secret-key uid: ac14043e-179c-11e6-af05-120592f139e9 type: Opaque

[helgi]

Does this consistently happen? Can you cat /app/deis/settings.py from that pod?

[jdumars]

NAMESPACE   NAME                          READY     STATUS             RESTARTS   AGE
deis        deis-controller-wxsi8         0/1       CrashLoopBackOff   11         13m

I can't seem to get anything out of the pod. It just says:

error: error executing remote command: Error executing command in container: container not found ("deis-controller")

Other pods exec fine, for example:

kubectl exec deis-workflow-manager-5imes --namespace=deis cat /etc/hosts --namespace=deis

Kubernetes-managed hosts file.

127.0.0.1 localhost

[helgi]

Have you observed this again?

[helgi]

I spoke with @jdumars and he can't reproduce this anymore. Going to close this until we we can reliably reproduce it