canonical / bundle-kubeflow

Charmed Kubeflow
Apache License 2.0
103 stars 49 forks source link

AttributeError: 'MySQLClient' object has no attribute 'root_password' #187

Closed rs22 closed 4 years ago

rs22 commented 4 years ago

This is a new issue for my problem at the bottom of #186.

python3 scripts/cli.py deploy-to uk8s --build reproducibly fails with

+ juju wait -wv -m kubeflow
DEBUG:root:katib-manager/0 workload status is error since 2020-03-25 22:20:41+00:00
ERROR:root:katib-manager/0 failed: workload status is error
Command '('juju', 'wait', '-wv', '-m', 'kubeflow')' returned non-zero exit status 1.

microk8s.kubectl logs -n kubeflow --tail 200 -l juju-operator=katib-manager prints this error:

2020-03-25 22:26:03 INFO juju-log mysql:9: Invoking reactive handler: reactive/katib_manager.py:21:start_charm
2020-03-25 22:26:03 INFO juju-log mysql:9: status-set: maintenance: configuring container
2020-03-25 22:26:03 ERROR juju-log mysql:9: Hook error:
Traceback (most recent call last):
  File "lib/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "lib/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "lib/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "lib/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-katib-manager-0/charm/reactive/katib_manager.py", line 48, in start_charm
    'DB_PASSWORD': mysql.root_password(),
AttributeError: 'MySQLClient' object has no attribute 'root_password'

2020-03-25 22:26:03 DEBUG mysql-relation-changed Traceback (most recent call last):
2020-03-25 22:26:03 DEBUG mysql-relation-changed   File "/var/lib/juju/agents/unit-katib-manager-0/charm/hooks/mysql-relation-changed", line 19, in <module>
2020-03-25 22:26:03 DEBUG mysql-relation-changed     main()
2020-03-25 22:26:03 DEBUG mysql-relation-changed   File "lib/charms/reactive/__init__.py", line 74, in main
2020-03-25 22:26:03 DEBUG mysql-relation-changed     bus.dispatch(restricted=restricted_mode)
2020-03-25 22:26:03 DEBUG mysql-relation-changed   File "lib/charms/reactive/bus.py", line 390, in dispatch
2020-03-25 22:26:03 DEBUG mysql-relation-changed     _invoke(other_handlers)
2020-03-25 22:26:03 DEBUG mysql-relation-changed   File "lib/charms/reactive/bus.py", line 359, in _invoke
2020-03-25 22:26:03 DEBUG mysql-relation-changed     handler.invoke()
2020-03-25 22:26:03 DEBUG mysql-relation-changed   File "lib/charms/reactive/bus.py", line 181, in invoke
2020-03-25 22:26:03 DEBUG mysql-relation-changed     self._action(*args)
2020-03-25 22:26:03 DEBUG mysql-relation-changed   File "/var/lib/juju/agents/unit-katib-manager-0/charm/reactive/katib_manager.py", line
48, in start_charm
2020-03-25 22:26:03 DEBUG mysql-relation-changed     'DB_PASSWORD': mysql.root_password(),
2020-03-25 22:26:03 DEBUG mysql-relation-changed AttributeError: 'MySQLClient' object has no attribute 'root_password'
2020-03-25 22:26:03 ERROR juju.worker.uniter.operation runhook.go:132 hook "mysql-relation-changed" failed: exit status 1

I can confirm that the password overlay file is properly generated during setup. I think it has something to do with the mysql interface (or the mariadb-k8s charm? -- I'm quite new to all this), since for metadata-api, which also requires the mysql interface, a similar error occurs:

2020-03-25 22:26:24 INFO juju-log mysql:12: Invoking reactive handler: reactive/metadata_api.py:26:start_charm
2020-03-25 22:26:24 INFO juju-log mysql:12: status-set: maintenance: configuring container
2020-03-25 22:26:24 ERROR juju-log mysql:12: Hook error:
Traceback (most recent call last):
  File "lib/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "lib/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "lib/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "lib/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-metadata-api-0/charm/reactive/metadata_api.py", line 50, in start_charm
    f"--mysql_service_password={mysql.root_password()}",
AttributeError: 'MySQLClient' object has no attribute 'root_password'

2020-03-25 22:26:24 DEBUG mysql-relation-changed Traceback (most recent call last):
2020-03-25 22:26:24 DEBUG mysql-relation-changed   File "/var/lib/juju/agents/unit-metadata-api-0/charm/hooks/mysql-relation-changed", line 19, in <module>
2020-03-25 22:26:24 DEBUG mysql-relation-changed     main()
2020-03-25 22:26:24 DEBUG mysql-relation-changed   File "lib/charms/reactive/__init__.py", line 74, in main
2020-03-25 22:26:24 DEBUG mysql-relation-changed     bus.dispatch(restricted=restricted_mode)
2020-03-25 22:26:24 DEBUG mysql-relation-changed   File "lib/charms/reactive/bus.py", line 390, in dispatch
2020-03-25 22:26:24 DEBUG mysql-relation-changed     _invoke(other_handlers)
2020-03-25 22:26:24 DEBUG mysql-relation-changed   File "lib/charms/reactive/bus.py", line 359, in _invoke
2020-03-25 22:26:24 DEBUG mysql-relation-changed     handler.invoke()
2020-03-25 22:26:24 DEBUG mysql-relation-changed   File "lib/charms/reactive/bus.py", line 181, in invoke
2020-03-25 22:26:24 DEBUG mysql-relation-changed     self._action(*args)
2020-03-25 22:26:24 DEBUG mysql-relation-changed   File "/var/lib/juju/agents/unit-metadata-api-0/charm/reactive/metadata_api.py", line 50, in start_charm
2020-03-25 22:26:24 DEBUG mysql-relation-changed     f"--mysql_service_password={mysql.root_password()}",
2020-03-25 22:26:24 DEBUG mysql-relation-changed AttributeError: 'MySQLClient' object has no attribute 'root_password'
2020-03-25 22:26:24 ERROR juju.worker.uniter.operation runhook.go:132 hook "mysql-relation-changed" failed: exit status 1

I initially suspected that this could have something to do with my prior experiments, which might have polluted some build cache and hence an incompatible version of mariadb-k8s is being used. I tried to clean up the system state by re-installing the juju snaps (snap remove --purge, ...) and removed the /tmp/charm-builds and ~/.cache/charm directories, but the problem persists.

rs22 commented 4 years ago

I could also reproduce the problem in a fresh multipass VM.

knkski commented 4 years ago

@rs22: Sorry, responding to this issue fell through the cracks, but this should now be fixed