canonical / mysql-operator

Machine charm for MySQL following the operator framework
https://charmhub.io/mysql
Apache License 2.0
7 stars 12 forks source link

Mysql fails with Access denied for user \'clusteradmin\'@\'10.246.164.206\' #121

Closed Sponge-Bas closed 1 year ago

Sponge-Bas commented 1 year ago

I'm trying to use the mysql charm instead of the mysql-innodb-cluster charm for an OpenStack deployment, but I'm running into a couple of issues:

Steps to reproduce

Following the examples from https://charmhub.io/mysql:

The docs suggest:

# Deploy the relevant charms
juju deploy -n 3 mysql --channel edge
juju deploy mycharm
# Relate mysql-router with keystone <- this implies that keystone should work here
juju relate mycharm:database mysql:database

I ran:

:~/ juju deploy keystone --channel yoga/stable --series jammy
:~/ juju deploy mysql --channel edge
:~/ juju relate keystone:database mysql:database
ERROR application "keystone" has no "database" relation

I assume that this is because there is still some work to do to make keystone compatible with the mysql charm.

Then I tried one of the legacy relations. I ran:

:~/ juju deploy keystone --channel yoga/stable --series jammy
:~/ juju deploy mysql --channel edge
:~/ juju deploy mysql-router keystone-mysql-router --channel 8.0/stable
:~/ juju relate keystone-mysql-router:db-router mysql:db-router
:~/ juju relate keystone:shared-db keystone-mysql-router:shared-db

Expected behavior

I would expect mysql to set up the database and allow keystone to connect to it.

Actual behavior

The deployment went into the following status:

:~/project$ juju status --relations
Model      Controller        Cloud/Region        Version  SLA          Timestamp
openstack  foundations-maas  maas_cloud/default  2.9.38   unsupported  14:16:53Z

App                    Version  Status   Scale  Charm         Channel      Rev  Exposed  Message
keystone               21.0.0   waiting      1  keystone      yoga/stable  590  no       Allowed_units list provided but this unit not present
keystone-mysql-router  8.0.32   waiting      1  mysql-router  8.0/stable    35  no       'db-router' incomplete, MySQL Router not yet bootstrapped
mysql                           error        1  mysql         edge         109  no       hook failed: "db-router-relation-changed"

Unit                        Workload  Agent  Machine  Public address  Ports     Message
keystone/1*                 blocked   idle   2        10.246.165.34   5000/tcp  Incomplete relations: database, Allowed_units list provided but this unit not present
  keystone-mysql-router/0*  waiting   idle            10.246.165.34             'db-router' incomplete, MySQL Router not yet bootstrapped
mysql/0*                    error     idle   1        10.246.164.206            hook failed: "db-router-relation-changed" for keystone-mysql-router:db-router

Machine  State    Address         Inst id   Series  AZ     Message
1        started  10.246.164.206  vault1-2  jammy   zone3  Deployed
2        started  10.246.165.34   vault1-1  jammy   zone2  Deployed

Relation provider                Requirer                         Interface     Type         Message
keystone-mysql-router:shared-db  keystone:shared-db               mysql-shared  subordinate
keystone:cluster                 keystone:cluster                 keystone-ha   peer
mysql:database-peers             mysql:database-peers             mysql-peers   peer
mysql:db-router                  keystone-mysql-router:db-router  mysql-router  regular
mysql:restart                    mysql:restart                    rolling_op    peer

Versions

see above

Log output

juju debug-log --replay ```shell Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-0/charm/src/mysql_vm_helpers.py", line 187, in _run_mysqlsh_script return subprocess.check_output( File "/usr/lib/python3.10/subprocess.py", line 420, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "/usr/lib/python3.10/subprocess.py", line 524, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/snap/bin/mysqlsh', '--no-wizard', '--python', '-f', '/root/snap/mysql-shell/common/tmp_r6gwjds']' returned non-zero exit status 1. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/mysql/v0/mysql.py", line 966, in get_cluster_primary_address output = self._run_mysqlsh_script("\n".join(get_cluster_primary_commands)) File "/var/lib/juju/agents/unit-mysql-0/charm/src/mysql_vm_helpers.py", line 191, in _run_mysqlsh_script raise MySQLClientError(e.stderr) charms.mysql.v0.mysql.MySQLClientError: b'Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory\nTraceback (most recent call last):\n File "", line 1, in \nmysqlsh.DBError: MySQL Error (1045): Shell.connect: Access denied for user \'clusteradmin\'@\'10.246.164.206\' (using password: YES)\n' unit-mysql-0: 14:09:53 ERROR unit.mysql/0.juju-log db-router:5: Uncaught exception while in charm code: Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-0/charm/src/mysql_vm_helpers.py", line 187, in _run_mysqlsh_script return subprocess.check_output( File "/usr/lib/python3.10/subprocess.py", line 420, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "/usr/lib/python3.10/subprocess.py", line 524, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/snap/bin/mysqlsh', '--no-wizard', '--python', '-f', '/root/snap/mysql-shell/common/tmp_r6gwjds']' returned non-zero exit status 1. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/mysql/v0/mysql.py", line 966, in get_cluster_primary_address output = self._run_mysqlsh_script("\n".join(get_cluster_primary_commands)) File "/var/lib/juju/agents/unit-mysql-0/charm/src/mysql_vm_helpers.py", line 191, in _run_mysqlsh_script raise MySQLClientError(e.stderr) charms.mysql.v0.mysql.MySQLClientError: b'Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory\nTraceback (most recent call last):\n File "", line 1, in \nmysqlsh.DBError: MySQL Error (1045): Shell.connect: Access denied for user \'clusteradmin\'@\'10.246.164.206\' (using password: YES)\n' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-0/charm/./src/charm.py", line 653, in main(MySQLOperatorCharm) File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/main.py", line 435, in main _emit_charm_event(charm, dispatcher.event_name) File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/main.py", line 144, in _emit_charm_event event_to_emit.emit(*args, **kwargs) File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/framework.py", line 355, in emit framework._emit(event) # noqa File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/framework.py", line 824, in _emit self._reemit(event_path) File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/framework.py", line 899, in _reemit custom_handler(event) File "/var/lib/juju/agents/unit-mysql-0/charm/src/relations/db_router.py", line 223, in _on_db_router_relation_changed requested_user_passwords, requested_user_applications = self._create_requested_users( File "/var/lib/juju/agents/unit-mysql-0/charm/src/relations/db_router.py", line 144, in _create_requested_users self.charm._mysql.configure_mysqlrouter_user( File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/mysql/v0/mysql.py", line 355, in configure_mysqlrouter_user primary_address = self.get_cluster_primary_address() File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/mysql/v0/mysql.py", line 969, in get_cluster_primary_address raise MySQLGetClusterPrimaryAddressError(e.message) charms.mysql.v0.mysql.MySQLGetClusterPrimaryAddressError: b'Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory\nTraceback (most recent call last):\n File "", line 1, in \nmysqlsh.DBError: MySQL Error (1045): Shell.connect: Access denied for user \'clusteradmin\'@\'10.246.164.206\' (using password: YES)\n' ```
taurus-forever commented 1 year ago

Dear @Basdbruijne , tnx for trying our operator! The README was a way outdated, new PR created to address it. Meanwhile, check README in PR.

The charm documentation is something we are working this sprint, please pay your attention to the tutorial in docs folder (also it is being actively updated on discourse).

TL;DR please try the following on your side (note db-router relational there):

juju deploy mysql --channel edge
juju deploy mysql-router --series focal
juju deploy keystone --series focal
juju relate mysql-router keystone
juju relate mysql:db-router mysql-router:db-router

it does the job for me:

Model    Controller           Cloud/Region         Version  SLA          Timestamp
default  localhost-localhost  localhost/localhost  2.9.38   unsupported  15:32:09+01:00

App           Version          Status  Scale  Charm         Channel     Rev  Exposed  Message
keystone      17.0.1           active      1  keystone      stable      539  no       Application Ready
mysql         8.0.32-0ubun...  active      1  mysql         edge        111  no       Unit is ready: Mode: RW
mysql-router  8.0.32           active      1  mysql-router  8.0/stable   35  no       Unit is ready

Unit               Workload  Agent  Machine  Public address  Ports     Message
keystone/0*        active    idle   1        10.158.123.127  5000/tcp  Unit is ready
  mysql-router/0*  active    idle            10.158.123.127            Unit is ready
mysql/0*           active    idle   0        10.158.123.115            Unit is ready: Mode: RW

Machine  State    Address         Inst id        Series  AZ  Message
0        started  10.158.123.115  juju-5ae2d0-0  jammy       Running
1        started  10.158.123.127  juju-5ae2d0-1  focal       Running

Your example also worked on my side:

Model    Controller           Cloud/Region         Version  SLA          Timestamp
default  localhost-localhost  localhost/localhost  2.9.38   unsupported  17:57:07+01:00

App                    Version          Status  Scale  Charm         Channel      Rev  Exposed  Message
keystone               21.0.0           active      1  keystone      yoga/stable  595  no       Application Ready
keystone-mysql-router  8.0.32           active      1  mysql-router  8.0/stable    35  no       Unit is ready
mysql                  8.0.32-0ubun...  active      1  mysql         edge         111  no       Unit is ready: Mode: RW

Unit                        Workload  Agent  Machine  Public address  Ports     Message
keystone/0*                 active    idle   0        10.158.123.90   5000/tcp  Unit is ready
  keystone-mysql-router/0*  active    idle            10.158.123.90             Unit is ready
mysql/0*                    active    idle   1        10.158.123.243            Unit is ready: Mode: RW

Machine  State    Address         Inst id        Series  AZ  Message
0        started  10.158.123.90   juju-412c71-0  jammy       Running
1        started  10.158.123.243  juju-412c71-1  jammy       Running

Relation provider                Requirer                         Interface     Type         Message
keystone-mysql-router:shared-db  keystone:shared-db               mysql-shared  subordinate  
keystone:cluster                 keystone:cluster                 keystone-ha   peer         
mysql:database-peers             mysql:database-peers             mysql-peers   peer         
mysql:db-router                  keystone-mysql-router:db-router  mysql-router  regular      
mysql:restart                    mysql:restart                    rolling_op    peer         

Is it still reproducible on your side? I see slightly different versions for charms. Tnx!

Sponge-Bas commented 1 year ago

I tried your config again, but the result is the same, from the debug-log:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-0/charm/src/mysql_vm_helpers.py", line 187, in _run_mysqlsh_script
    return subprocess.check_output(
  File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/snap/bin/mysqlsh', '--no-wizard', '--python', '-f', '/root/snap/mysql-shell/common/tmpul_7exv4']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/mysql/v0/mysql.py", line 989, in get_cluster_primary_address
    output = self._run_mysqlsh_script("\n".join(get_cluster_primary_commands))
  File "/var/lib/juju/agents/unit-mysql-0/charm/src/mysql_vm_helpers.py", line 191, in _run_mysqlsh_script
    raise MySQLClientError(e.stderr)
charms.mysql.v0.mysql.MySQLClientError: b'Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory\nTraceback (most recent call last):\n  File "<string>", line 1, in <module>\nmysqlsh.DBError: MySQL Error (1045): Shell.connect: Access denied for user \'clusteradmin\'@\'10.246.167.63\' (using password: YES)\n'
unit-mysql-0: 20:28:56 ERROR unit.mysql/0.juju-log db-router:4: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-0/charm/src/mysql_vm_helpers.py", line 187, in _run_mysqlsh_script
    return subprocess.check_output(
  File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/snap/bin/mysqlsh', '--no-wizard', '--python', '-f', '/root/snap/mysql-shell/common/tmpul_7exv4']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/mysql/v0/mysql.py", line 989, in get_cluster_primary_address
    output = self._run_mysqlsh_script("\n".join(get_cluster_primary_commands))
  File "/var/lib/juju/agents/unit-mysql-0/charm/src/mysql_vm_helpers.py", line 191, in _run_mysqlsh_script
    raise MySQLClientError(e.stderr)
charms.mysql.v0.mysql.MySQLClientError: b'Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory\nTraceback (most recent call last):\n  File "<string>", line 1, in <module>\nmysqlsh.DBError: MySQL Error (1045): Shell.connect: Access denied for user \'clusteradmin\'@\'10.246.167.63\' (using password: YES)\n'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-0/charm/./src/charm.py", line 653, in <module>
    main(MySQLOperatorCharm)
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/main.py", line 435, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/main.py", line 144, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/framework.py", line 355, in emit
    framework._emit(event)  # noqa
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/framework.py", line 824, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/framework.py", line 899, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-mysql-0/charm/src/relations/db_router.py", line 223, in _on_db_router_relation_changed
    requested_user_passwords, requested_user_applications = self._create_requested_users(
  File "/var/lib/juju/agents/unit-mysql-0/charm/src/relations/db_router.py", line 144, in _create_requested_users
    self.charm._mysql.configure_mysqlrouter_user(
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/mysql/v0/mysql.py", line 363, in configure_mysqlrouter_user
    primary_address = self.get_cluster_primary_address()
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/mysql/v0/mysql.py", line 992, in get_cluster_primary_address
    raise MySQLGetClusterPrimaryAddressError(e.message)
charms.mysql.v0.mysql.MySQLGetClusterPrimaryAddressError: b'Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory\nTraceback (most recent call last):\n  File "<string>", line 1, in <module>\nmysqlsh.DBError: MySQL Error (1045): Shell.connect: Access denied for user \'clusteradmin\'@\'10.246.167.63\' (using password: YES)\n'

The difference between our experiments is that I am using a maas controller while you are using a localhost controller. After switching to a localhost controller I did indeed get the same result as you.

Judging by the Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory message, I think there is some initialization missing on Maas controllers but that is set by default on lxd controllers.

Do you have access to a Maas controller for testing? Otherwise, we can arrange some temporary access to the SQA lab.

taurus-forever commented 1 year ago

Dear @Basdbruijne, Yes, we need a troubleshooting session together on MaaS. We have planned this ticket to the next pulse, please ping @paulomach in MM the next week to trace it together. Tnx!

paulomach commented 1 year ago

The root cause is that the hostname name resolution is resolving to 127.0.1.1 which is not bound to any interface, and mysqld will not accepted as a valid host to configure as a Group Replication node.

Setting the report_host variable in static configuration file like done for k8s, will solve the issue.

This is an MAAS issue, as described here and will happen for ephemeral environment.

paulomach commented 1 year ago
Every 1.0s: juju status --color --relations                                                                                                                                                                                                 lab3-anorith-cpe-bdc275ba-1a63-4397-8fcb-2d66e8cb278f: Tue Mar 14 19:08:44 2023

Model       Controller        Cloud/Region  Version  SLA          Timestamp
test-mysql  foundations-maas  maas_cloud    2.9.42   unsupported  19:08:44Z

App                    Version  Status  Scale  Charm         Channel      Rev  Exposed  Message
keystone               21.0.0   active      1  keystone      yoga/stable  595  no       Application Ready
keystone-mysql-router  8.0.32   active      1  mysql-router  8.0/stable    35  no       Unit is ready
mysql                  8.0.32   active      1  mysql                        0  no       Primary

Unit                        Workload  Agent  Machine  Public address  Ports               Message
keystone/0*                 active    idle   1        10.244.40.105   5000/tcp            Unit is ready
  keystone-mysql-router/0*  active    idle            10.244.40.105                       Unit is ready
mysql/1*                    active    idle   2        10.244.40.106   3306/tcp,33060/tcp  Primary

Machine  State    Address        Inst id  Series  AZ     Message
1        started  10.244.40.105  vault-1  jammy   zone1  Deployed
2        started  10.244.40.106  vault-3  jammy   zone3  Deployed

Relation provider                Requirer                         Interface     Type         Message
keystone-mysql-router:shared-db  keystone:shared-db               mysql-shared  subordinate
keystone:cluster                 keystone:cluster                 keystone-ha   peer
mysql:database-peers             mysql:database-peers             mysql_peers   peer
mysql:db-router                  keystone-mysql-router:db-router  mysql-router  regular
mysql:restart                    mysql:restart                    rolling_op    peer

Got a fix running. PR on the way