search

canonical / mysql-operator

Machine charm for MySQL following the operator framework
https://charmhub.io/mysql
Apache License 2.0
7 stars 12 forks source link

DPE-2455 Fix bug that caused unnecessary truncation of the mysql hosts cache #309

Closed shayancanonical closed 1 year ago

shayancanonical commented 1 year ago

Issue

  1. We are using info debug messages which is causing unnecessary volume in the debug log
  2. We are not return-ing in _potentially_update_etc_hosts if there are no changes to the /etc/hosts file
  3. When a unit has departed, in the handler, we are not checking if an update to the etc hosts file is needed. If the file was updated due to the relation changed event, we are repeating an update of the etc hosts file and unnecessarily truncating the mysql hosts cache

Solution

  1. Use debug statements
  2. Add return statements
  3. Check if the departing unit is in the etc hosts file, and only then run the logic in _remove_host_from_etc_hosts

Logs from sample removal of unit from a cluster of 3 nodes:

unit-mysql-2: 13:55:35 DEBUG unit.mysql/2.juju-log ops 2.5.1 up and running.
unit-mysql-1: 13:55:35 DEBUG unit.mysql/1.juju-log database-peers:0: ops 2.5.1 up and running.
unit-mysql-1: 13:55:35 INFO unit.mysql/1.juju-log database-peers:0: Starting IP address observer process
unit-mysql-2: 13:55:35 DEBUG unit.mysql/2.juju-log Emitting Juju event database_storage_detaching.
unit-mysql-0: 13:55:35 DEBUG unit.mysql/0.juju-log database-peers:0: ops 2.5.1 up and running.
unit-mysql-1: 13:55:35 INFO unit.mysql/1.juju-log database-peers:0: Started IP address observer process with PID <Popen: returncode: None args: ['/usr/bin/python3', 'src/ip_address_observer...>
unit-mysql-2: 13:55:35 DEBUG unit.mysql/2.juju-log Checking existence of unit mysql-2 in cluster cluster-37de9aa812d1a67a0119c06f017a7428
unit-mysql-1: 13:55:35 DEBUG unit.mysql/1.juju-log database-peers:0: Emitting Juju event database_peers_relation_departed.
unit-mysql-0: 13:55:35 DEBUG unit.mysql/0.juju-log database-peers:0: Emitting Juju event database_peers_relation_departed.
unit-mysql-0: 13:55:35 DEBUG unit.mysql/0.juju-log database-peers:0: Checking if an entry for mysql/2 is in /etc/hosts
unit-mysql-0: 13:55:35 DEBUG unit.mysql/0.juju-log database-peers:0: Removing entry for mysql/2 from /etc/hosts
unit-mysql-1: 13:55:35 DEBUG unit.mysql/1.juju-log database-peers:0: Checking if an entry for mysql/2 is in /etc/hosts
unit-mysql-1: 13:55:35 DEBUG unit.mysql/1.juju-log database-peers:0: Removing entry for mysql/2 from /etc/hosts
unit-mysql-0: 13:55:35 DEBUG unit.mysql/0.juju-log database-peers:0: Truncating the MySQL host cache
unit-mysql-1: 13:55:35 DEBUG unit.mysql/1.juju-log database-peers:0: Truncating the MySQL host cache
unit-mysql-2: 13:55:36 DEBUG unit.mysql/2.juju-log Getting cluster primary member's address from 10.105.138.163
unit-mysql-0: 13:55:37 DEBUG unit.mysql/0.juju-log upgrade:1: ops 2.5.1 up and running.
unit-mysql-1: 13:55:37 DEBUG unit.mysql/1.juju-log upgrade:1: ops 2.5.1 up and running.
unit-mysql-2: 13:55:38 DEBUG unit.mysql/2.juju-log Attempting to acquire lock unit-teardown on juju-f91bfd-0.lxd for unit mysql-2
unit-mysql-0: 13:55:38 DEBUG unit.mysql/0.juju-log upgrade:1: Emitting Juju event upgrade_relation_departed.
unit-mysql-1: 13:55:38 DEBUG unit.mysql/1.juju-log upgrade:1: Emitting Juju event upgrade_relation_departed.
unit-mysql-2: 13:55:38 DEBUG unit.mysql/2.juju-log Getting cluster member addresses, excluding units ['mysql-2']
unit-mysql-0: 13:55:39 DEBUG unit.mysql/0.juju-log database-peers:0: ops 2.5.1 up and running.
unit-mysql-0: 13:55:39 DEBUG unit.mysql/0.juju-log database-peers:0: Emitting Juju event database_peers_relation_changed.
unit-mysql-0: 13:55:39 DEBUG unit.mysql/0.juju-log database-peers:0: No hostnames in /etc/hosts changed. Skipping update to /etc/hosts
unit-mysql-2: 13:55:40 DEBUG unit.mysql/2.juju-log Removing instance 10.105.138.163 from cluster cluster-37de9aa812d1a67a0119c06f017a7428
unit-mysql-2: 13:55:44 DEBUG unit.mysql/2.juju-log Getting cluster primary member's address from juju-f91bfd-0.lxd:3306
unit-mysql-2: 13:55:44 DEBUG unit.mysql/2.juju-log Releasing lock unit-teardown on juju-f91bfd-0.lxd for unit mysql-2
unit-mysql-2: 13:55:46 DEBUG unit.mysql/2.juju-log database-peers:0: ops 2.5.1 up and running.
unit-mysql-2: 13:55:46 DEBUG unit.mysql/2.juju-log database-peers:0: Emitting Juju event database_peers_relation_departed.
unit-mysql-2: 13:55:46 DEBUG unit.mysql/2.juju-log database-peers:0: Checking if an entry for mysql/0 is in /etc/hosts
unit-mysql-2: 13:55:46 DEBUG unit.mysql/2.juju-log database-peers:0: Removing entry for mysql/0 from /etc/hosts
unit-mysql-2: 13:55:46 DEBUG unit.mysql/2.juju-log database-peers:0: Truncating the MySQL host cache
unit-mysql-2: 13:55:47 DEBUG unit.mysql/2.juju-log upgrade:1: ops 2.5.1 up and running.
unit-mysql-2: 13:55:47 DEBUG unit.mysql/2.juju-log upgrade:1: Emitting Juju event upgrade_relation_departed.
unit-mysql-2: 13:55:48 DEBUG unit.mysql/2.juju-log database-peers:0: ops 2.5.1 up and running.
unit-mysql-2: 13:55:48 DEBUG unit.mysql/2.juju-log database-peers:0: Emitting Juju event database_peers_relation_departed.
unit-mysql-2: 13:55:48 DEBUG unit.mysql/2.juju-log database-peers:0: Checking if an entry for mysql/1 is in /etc/hosts
unit-mysql-2: 13:55:48 DEBUG unit.mysql/2.juju-log database-peers:0: Removing entry for mysql/1 from /etc/hosts
unit-mysql-2: 13:55:48 DEBUG unit.mysql/2.juju-log database-peers:0: Truncating the MySQL host cache
unit-mysql-2: 13:55:49 DEBUG unit.mysql/2.juju-log upgrade:1: ops 2.5.1 up and running.
unit-mysql-2: 13:55:49 DEBUG unit.mysql/2.juju-log upgrade:1: Emitting Juju event upgrade_relation_departed.
unit-mysql-2: 13:55:50 DEBUG unit.mysql/2.juju-log ops 2.5.1 up and running.
unit-mysql-2: 13:55:50 DEBUG unit.mysql/2.juju-log Emitting Juju event stop.
unit-mysql-2: 13:55:51 DEBUG unit.mysql/2.juju-log ops 2.5.1 up and running.
unit-mysql-2: 13:55:51 DEBUG unit.mysql/2.juju-log Emitting Juju event remove.
codecov[bot] commented 1 year ago

Codecov Report

Merging #309 (849fe53) into main (31fcb87) will decrease coverage by 0.25%. The diff coverage is 0.00%.

:exclamation: Current head 849fe53 differs from pull request most recent head 8d12426. Consider uploading reports for the commit 8d12426 to get more accurate results

@@            Coverage Diff             @@
##             main     #309      +/-   ##
==========================================
- Coverage   63.52%   63.27%   -0.25%     
==========================================
  Files          15       15              
  Lines        2832     2843      +11     
  Branches      368      372       +4     
==========================================
  Hits         1799     1799              
- Misses        916      927      +11     
  Partials      117      117
Files Changed Coverage Δ
src/hostname_resolution.py 34.23% <0.00%> (-3.77%) :arrow_down:
src/mysql_vm_helpers.py 62.94% <0.00%> (ø)