search

canonical / mysql-operator

Machine charm for MySQL following the operator framework
https://charmhub.io/mysql
Apache License 2.0
7 stars 10 forks source link

Strange behavior for vsphere deployments #162

Closed jamesbeedy closed 1 year ago

jamesbeedy commented 1 year ago

I get this message when deploying mysql from edge in a sphere cloud.

App           Version  Status   Scale  Charm         Channel   Rev  Exposed  Message
mysql                  blocked      1  mysql         edge      132  no       Failed to configure instance for InnoDB
mysql-router  8.0.32   blocked      1  mysql-router  8.0/edge   61  no       'db-router' incomplete, 'shared-db' missing, MySQL Router not yet bootstrapped
slurmctld     69a8126  blocked      1  slurmctld     edge       50  no       Machine needs reboot
slurmd        8800da8  blocked      1  slurmd        edge       60  no       Machine needs reboot
slurmdbd      b5b1f35  blocked      1  slurmdbd      edge       43  no       Machine needs reboot

Unit               Workload  Agent  Machine  Public address  Ports  Message
mysql/1*           blocked   idle   4        10.104.138.235         Failed to configure instance for InnoDB
slurmctld/0*       blocked   idle   1        10.104.138.19          Machine needs reboot
slurmd/0*          blocked   idle   0        10.104.139.16          Machine needs reboot
slurmdbd/0*        blocked   idle   2        10.104.138.230         Machine needs reboot
  mysql-router/0*  blocked   idle            10.104.138.230         'db-router' incomplete, 'shared-db' missing, MySQL Router not yet bootstrapped

The command I used: juju deploy mysql --channel edge --constraints "cores=2 mem=4G root-disk=20G" --series jammy

This is what I see in the logs:

unit-mysql-1: 14:45:12 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch)
unit-mysql-1: 14:45:12 INFO juju.worker.uniter found queued "start" hook
unit-mysql-1: 14:45:13 INFO unit.mysql/1.juju-log Running legacy hooks/start.
unit-mysql-1: 14:45:13 INFO unit.mysql/1.juju-log Retrieving the total memory of the server
unit-mysql-1: 14:45:20 ERROR unit.mysql/1.juju-log Failed to configure instance: 10.104.138.235 with error b'Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory\nConfiguring local MySQL instance listening at port 3306 for use in an InnoDB cluster...\n\nThis instance reports its own address as \x1b[1mjuju-d4ca05-4:3306\x1b[0m\nClients and other cluster members will communicate with it through this address by default. If this is not correct, the report_host MySQL system variable should be changed.\n\x1b[31mERROR: \x1b[0mCannot use host \'juju-d4ca05-4\' for instance \'juju-d4ca05-4:3306\' because it resolves to an IP address (127.0.1.1) that does not match a real network interface, thus it is not supported. Change your system settings and/or set the MySQL server \'report_host\' variable to a hostname that resolves to a supported IP address.\nTraceback (most recent call last):\n  File "<string>", line 1, in <module>\nRuntimeError: Dba.configure_instance: Invalid host/IP \'juju-d4ca05-4\' resolves to \'127.0.1.1\' which is not supported.\n\n'
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-1/charm/src/mysql_vm_helpers.py", line 493, in _run_mysqlsh_script
    return subprocess.check_output(
  File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['charmed-mysql.mysqlsh', '--no-wizard', '--python', '-f', '/var/snap/charmed-mysql/common/tmpz3k2lver']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-1/charm/lib/charms/mysql/v0/mysql.py", line 588, in configure_instance
    self._run_mysqlsh_script("\n".join(configure_instance_command))
  File "/var/lib/juju/agents/unit-mysql-1/charm/src/mysql_vm_helpers.py", line 497, in _run_mysqlsh_script
    raise MySQLClientError(e.stderr)
charms.mysql.v0.mysql.MySQLClientError: b'Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory\nConfiguring local MySQL instance listening at port 3306 for use in an InnoDB cluster...\n\nThis instance reports its own address as \x1b[1mjuju-d4ca05-4:3306\x1b[0m\nClients and other cluster members will communicate with it through this address by default. If this is not correct, the report_host MySQL system variable should be changed.\n\x1b[31mERROR: \x1b[0mCannot use host \'juju-d4ca05-4\' for instance \'juju-d4ca05-4:3306\' because it resolves to an IP address (127.0.1.1) that does not match a real network interface, thus it is not supported. Change your system settings and/or set the MySQL server \'report_host\' variable to a hostname that resolves to a supported IP address.\nTraceback (most recent call last):\n  File "<string>", line 1, in <module>\nRuntimeError: Dba.configure_instance: Invalid host/IP \'juju-d4ca05-4\' resolves to \'127.0.1.1\' which is not supported.\n\n'

Deploying on focal gives an different error

unit-mysql-0: 14:39:51 INFO juju.worker.uniter awaiting error resolution for "storage-attached" hook
unit-mysql-0: 14:39:51 WARNING unit.mysql/0.database-storage-attached /usr/lib/python3/dist-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.15) or chardet (3.0.4) doesn't match a supported version!
unit-mysql-0: 14:39:51 WARNING unit.mysql/0.database-storage-attached   warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
unit-mysql-0: 14:39:51 WARNING unit.mysql/0.database-storage-attached Traceback (most recent call last):
unit-mysql-0: 14:39:51 WARNING unit.mysql/0.database-storage-attached   File "./src/charm.py", line 23, in <module>
unit-mysql-0: 14:39:51 WARNING unit.mysql/0.database-storage-attached     from charms.mysql.v0.tls import MySQLTLS
unit-mysql-0: 14:39:51 WARNING unit.mysql/0.database-storage-attached   File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/mysql/v0/tls.py", line 30, in <module>
unit-mysql-0: 14:40:31 INFO juju.worker.uniter awaiting error resolution for "storage-attached" hook
unit-mysql-0: 14:40:31 WARNING unit.mysql/0.database-storage-attached /usr/lib/python3/dist-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.15) or chardet (3.0.4) doesn't match a supported version!
unit-mysql-0: 14:40:31 WARNING unit.mysql/0.database-storage-attached   warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
unit-mysql-0: 14:40:31 WARNING unit.mysql/0.database-storage-attached Traceback (most recent call last):
unit-mysql-0: 14:40:31 WARNING unit.mysql/0.database-storage-attached   File "./src/charm.py", line 23, in <module>
unit-mysql-0: 14:40:31 WARNING unit.mysql/0.database-storage-attached     from charms.mysql.v0.tls import MySQLTLS
unit-mysql-0: 14:40:31 WARNING unit.mysql/0.database-storage-attached   File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/mysql/v0/tls.py", line 30, in <module>
unit-mysql-0: 14:40:31 WARNING unit.mysql/0.database-storage-attached     from charms.tls_certificates_interface.v1.tls_certificates import (
unit-mysql-0: 14:40:31 WARNING unit.mysql/0.database-storage-attached   File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/tls_certificates_interface/v1/tls_certificates.py", line 257, in <module>
unit-mysql-0: 14:40:31 WARNING unit.mysql/0.database-storage-attached     from cryptography import x509
unit-mysql-0: 14:40:31 WARNING unit.mysql/0.database-storage-attached   File "/var/lib/juju/agents/unit-mysql-0/charm/venv/cryptography/x509/__init__.py", line 6, in <module>
unit-mysql-0: 14:40:31 WARNING unit.mysql/0.database-storage-attached     from cryptography.x509 import certificate_transparency
unit-mysql-0: 14:40:31 WARNING unit.mysql/0.database-storage-attached   File "/var/lib/juju/agents/unit-mysql-0/charm/venv/cryptography/x509/certificate_transparency.py", line 10, in <module>
unit-mysql-0: 14:40:31 WARNING unit.mysql/0.database-storage-attached     from cryptography.hazmat.bindings._rust import x509 as rust_x509
unit-mysql-0: 14:40:31 WARNING unit.mysql/0.database-storage-attached ImportError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /var/lib/juju/agents/unit-mysql-0/charm/venv/cryptography/hazmat/bindings/_rust.abi3.so)

This charm seems to deploy just fine in my local lxd environment .... but fails on vsphere.

How could I investigate this further?

jamesbeedy commented 1 year ago

@kwmonroe comments https://github.com/juju-solutions/charm-mysql/issues/44#issuecomment-1482191690

taurus-forever commented 1 year ago

Dear James,

Thank you for the bugreport! We will check it and come back. At the moment it is a first bugreport about usage vsphere and I suspect something similar to recently addressed MaaS issue: https://github.com/canonical/mysql-operator/issues/121. Please compare this and this:

The root cause is that the hostname name resolution is resolving to 127.0.1.1 which is not bound to any interface, and mysqld will not accepted as a valid host to configure as a Group Replication node.

VS

... my vsphere hostname resolved to loopback and my aws unit to a valid ip on a real interface.

At the moment mysql charm is in active migration process. The stable and edge risks have completely different codebase (reactive charm VS operators framework (OF)).

Let's continue discussion here for the OF charm in edge risk. You are correct, we have two series there focal and jammy. Please focus on jammy and do NOT use focal as we do not update focal for a while(focal revision 86, jammy revision 133).

Hope this will bring you a bit more light to the situation.

paulomach commented 1 year ago

Hello @jamesbeedy ,

there's a proposed fix on PR#170 In the meantime, the proposed fix is also published to a temporary charmhub branch (expire 2023-04-27), that you can deploy with:

juju deploy mysql --channel edge/vsphere --series jammy
jamesbeedy commented 5 months ago

Just great! Thank you @paulomach!