search

canonical / mysql-operator

Machine charm for MySQL following the operator framework
https://charmhub.io/mysql
Apache License 2.0
7 stars 10 forks source link

Deployment failed on MAAS: Cannot use host 'fun-tick.maas' for instance 'fun-tick.maas:3306' because it resolves to an IP address (127.0.1.1). #366

Closed xtrusia closed 4 months ago

xtrusia commented 9 months ago

Steps to reproduce

  1. I faced this issue just when I deploy mysql in maas based environment.

Expected behavior

deployed well

Actual behavior

Failed to configure instance for InnoDB

Versions

Operating system:

Juju CLI: 2.9.42

Juju agent: 2.9.42

Charm revision: 8.0/stable 196, 8.0/edge 203

LXD:

Log output

Juju debug log:

2023-12-07 02:01:52 ERROR unit.mysql/0.juju-log server.go:316 Failed to configure instance: node-21.maas with error b'Configuring local MySQL instance listening at port 3306 for use in an InnoDB cluster...\n\nThis instance reports its own address as \x1b[1mnode-21.maas:3306\x1b[0m\n\x1b[31mERROR: \x1b[0mCannot use host \'node-21.maas\' for instance \'node-21.maas:3306\' because it resolves to an IP address (127.0.1.1) that does not match a real network interface, thus it is not supported. Change your system settings and/or set the MySQL server \'report_host\' variable to a hostname that resolves to a supported IP address.\nTraceback (most recent call last):\n File "", line 1, in \nRuntimeError: Dba.configure_instance: Invalid host/IP \'node-21.maas\' resolves to \'127.0.1.1\' which is not supported.\n\n' Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-0/charm/src/mysql_vm_helpers.py", line 605, in _run_mysqlsh_script return subprocess.check_output( File "/usr/lib/python3.10/subprocess.py", line 421, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "/usr/lib/python3.10/subprocess.py", line 526, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['charmed-mysql.mysqlsh', '--no-wizard', '--python', '-f', '/var/snap/charmed-mysql/common/tmppskk1yxg']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/mysql/v0/mysql.py", line 1156, in configure_instance self._run_mysqlsh_script("\n".join(configure_instance_command)) File "/var/lib/juju/agents/unit-mysql-0/charm/src/mysql_vm_helpers.py", line 609, in _run_mysqlsh_script raise MySQLClientError(e.stderr) charms.mysql.v0.mysql.MySQLClientError: b'Configuring local MySQL instance listening at port 3306 for use in an InnoDB cluster...\n\nThis instance reports its own address as \x1b[1mnode-21.maas:3306\x1b[0m\n\x1b[31mERROR: \x1b[0mCannot use host \'node-21.maas\' for instance \'node-21.maas:3306\' because it resolves to an IP address (127.0.1.1) that does not match a real network interface, thus it is not supported. Change your system settings and/or set the MySQL server \'report_host\' variable to a hostname that resolves to a supported IP address.\nTraceback (most recent call last):\n File "", line 1, in \nRuntimeError: Dba.configure_instance: Invalid host/IP \'node-21.maas\' resolves to \'127.0.1.1\' which is not supported.\n\n'

Additional context

github-actions[bot] commented 9 months ago

https://warthogs.atlassian.net/browse/DPE-3115

taurus-forever commented 9 months ago

@paulomach please comment it: ... Cannot use host 'node-21.maas' for instance 'node-21.maas:3306' because it resolves to an IP address (127.0.1.1) that does not match a real network interface, thus it is not supported ...

paulomach commented 9 months ago

@paulomach please comment it: ... Cannot use host 'node-21.maas' for instance 'node-21.maas:3306' because it resolves to an IP address (127.0.1.1) that does not match a real network interface, thus it is not supported ...

This was solved in the past, but I thinks there's a regression on the ip observer machinery not addressing 127.0.1.1 being present in /etc/hosts of ubuntu images.

@xtrusia , can you just confirm that there's a line like:

127.0.1.1   node-21

on these machines /etc/hosts file?

xtrusia commented 9 months ago

@paulomach please comment it: ... Cannot use host 'node-21.maas' for instance 'node-21.maas:3306' because it resolves to an IP address (127.0.1.1) that does not match a real network interface, thus it is not supported ...

This was solved in the past, but I thinks there's a regression on the ip observer machinery not addressing 127.0.1.1 being present in /etc/hosts of ubuntu images.

@xtrusia , can you just confirm that there's a line like:

127.0.1.1   node-21

on these machines /etc/hosts file?

I redeployed it with node-20 please refer to below hosts file

# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.debian.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
#     /etc/cloud/cloud.cfg or cloud-config from user-data
#
127.0.1.1 node-20.maas node-20
127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

10.0.0.210 node-20.maas node-20 # unit=mysql/0
paulomach commented 9 months ago

Thank you @xtrusia , as a workaround, if you remove the line:

127.0.1.1 node-20.maas node-20

the charm should be able to bootstrap. Generally, you can safely remove all 127.0.1.1 lines from the machines deploying mysql

nobuto-m commented 6 months ago

Is there a plan to fix this?

Long story, short, juju deploy mysql --channel 8.0/edge --base ubuntu@22.04 simply fails with MAAS provider, and logging into each node and update /etc/hosts is not a sustainable workaround to me.

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/mysql/v0/mysql.py", line 1125, in configure_instance
    self._run_mysqlsh_script("\n".join(configure_instance_command))
  File "/var/lib/juju/agents/unit-mysql-0/charm/src/mysql_vm_helpers.py", line 659, in _run_mysqlsh_script
    raise MySQLClientError(e.stderr)
charms.mysql.v0.mysql.MySQLClientError: b'Configuring local MySQL instance listening at port 3306 for use in an InnoDB cluster...\n\nThis instance reports its own address as \x1b[1mfun-tick.maas:3306\x1b[0m\n\x1b[31mERROR: \x1b[0mCannot use host \'fun-tick.maas\' for instance \'fun-tick.maas:3306\' because it resolves to an IP address (127.0.1.1) that does not match a real network interface, thus it is not supported. Change your system settings and/or set the MySQL server \'report_host\' variable to a hostname that resolves to a supported IP address.\nTraceback (most recent call last):\n  File "<string>", line 1, in <module>\nRuntimeError: Dba.configure_instance: Invalid host/IP \'fun-tick.maas\' resolves to \'127.0.1.1\' which is not supported.\n\n'
$ grep . /etc/hostname /etc/hosts
/etc/hostname:fun-tick
/etc/hosts:# Your system has configured 'manage_etc_hosts' as True.
/etc/hosts:# As a result, if you wish for changes to this file to persist
/etc/hosts:# then you will need to either
/etc/hosts:# a.) make changes to the master file in /etc/cloud/templates/hosts.debian.tmpl
/etc/hosts:# b.) change or remove the value of 'manage_etc_hosts' in
/etc/hosts:#     /etc/cloud/cloud.cfg or cloud-config from user-data
/etc/hosts:#
/etc/hosts:127.0.1.1 fun-tick.maas fun-tick
/etc/hosts:127.0.0.1 localhost
/etc/hosts:# The following lines are desirable for IPv6 capable hosts
/etc/hosts:::1 localhost ip6-localhost ip6-loopback
/etc/hosts:ff02::1 ip6-allnodes
/etc/hosts:ff02::2 ip6-allrouters
/etc/hosts:192.168.151.114 fun-tick.maas fun-tick # unit=mysql/0
jeffreychang911 commented 5 months ago

SolQA bumped into this issue as well, when deploying mysql 8.0/candidate or edge to maas_cloud.

taurus-forever commented 5 months ago

Reproduced it locally as well using MAAS dev setup: https://github.com/canonical/maas-dev-setup/issues/9

Anamnesis:

The current implementation is solution/fix/workaround for: a) MySQL Cluster is being deployed on hostnames due to IPs instabilities on day 2 lifetime. b) lack of hostnames support on Juju forced charm to handle /etc/hosts manually. c) It also fixes lack of units hostname resolution on openstack (for cross-server units).

@paulomach 1) at this stage, mysql charm went into blocked state, while IMHO the error state should be used (to allow users execute juju resolve mysql/0 after tuning /etc/host to continue bootstrap. 2) we most probably should stop using the hostname as it will be often outside of our control and move to report_host to something we define, e.g. juju-${app}-unit-${unit_id}:

127.0.1.1 fun-tick.maas fun-tick
127.0.0.1 localhost

192.168.151.114 juju-mysql-unit-0.local juju-mysql-unit-0 # unit=mysql/0

Otherwise we will continue facing issue when user/maas/openstack/VM/server wants to have special localhost/hostname setup... CC: @delgod we need your thoughts and sign for migration to report_host.

P.S. Is it a time to introduce .svc.cluster.local for VMs?

P.P.S. The manual workaround for MAAS (if you need one):

ubuntu@maas:~$ juju add-machine
created machine 5

# wait for machine fully deployed

ubuntu@maas:~$ juju ssh 5
ubuntu@honest-troll:~$ sudo vim /etc/hosts   # remove '127.0.1.1 honest-troll.maas ...' line
ubuntu@honest-troll:~$ sudo vim /etc/cloud/templates/hosts.debian.tmpl # remove '127.0.1.1 ...' line
ubuntu@honest-troll:~$ 
logout
Connection to 10.20.0.9 closed.

ubuntu@maas:~$ juju deploy mysql --channel 8.0/edge --base ubuntu@22.04 --to 5
Deployed "mysql" from charm-hub charm "mysql", revision 221 in channel 8.0/edge on ubuntu@22.04/stable
ubuntu@maas:~$ 

ubuntu@maas:~$ juju status
Model  Controller       Cloud/Region  Version  SLA          Timestamp
mysql  maas-controller  maas/default  3.4.2    unsupported  12:47:22+02:00

App         Version          Status   Scale  Charm           Channel        Rev  Exposed  Message
mysql       8.0.36-0ubun...  active       1  mysql           8.0/edge       221  no       

Unit        Workload  Agent  Machine  Public address  Ports           Message
mysql/0*    active    idle   5        10.20.0.9       3306,33060/tcp  Primary

Machine  State    Address    Inst id       Base          AZ       Message
5        started  10.20.0.9  honest-troll  ubuntu@22.04  default  Deployed
ubuntu@maas:~$
paulomach commented 4 months ago

@xtrusia , fix on release 232 available on channel 8.0/beta