search

canonical / mysql-router-operator

Mysql router operator charm
https://charmhub.io/mysql-router?channel=dpe/edge
Apache License 2.0
3 stars 3 forks source link

mysql-router unit goes into error state when removing the mysql-router application #114

Closed shayancanonical closed 1 week ago

shayancanonical commented 5 months ago

Steps to reproduce

  1. juju deploy -n 1 mysql --channel=8.0/stable
  2. juju deploy -n 1 mysql-test-app --channel=latest/stable
  3. juju deploy -n 1 mysql-router --channel=dpe/edge
  4. juju relate mysql-test-app mysql-router
  5. juju relate mysql-router mysql
  6. wait for all units to go into active status
  7. juju remove-application mysql-router ^ mysql-router unit goes into error state, halting the removal of the mysql-test-app unit as well

Expected behavior

mysql-router unit should exit cleanly, allowing the mysql-test-app to be removed cleanly as well

Actual behavior

image

Versions

Operating system: Ubuntu 22.04.3 LTS

Juju CLI: 3.4.0-genericlinux-amd64

Juju agent: 3.4.0

mysql charm revision: 196 mysql-router charm revision: 149

LXD: 5.20

Log output

Juju debug log: 

unit-mysql-test-app-0: 10:41:43 ERROR unit.mysql-test-app/0.juju-log database:6: Unable to query the database                                                                                                      
Traceback (most recent call last):                                                                                                                                                                                 
  File "/var/lib/juju/agents/unit-mysql-test-app-0/charm/venv/mysql/connector/connection_cext.py", line 291, in _open_connection                                                                                   
    self._cmysql.connect(**cnx_kwargs)                                                                                                                                                                             
_mysql_connector.MySQLInterfaceError: Can't connect to local MySQL server through socket '/var/snap/charmed-mysql/common/run/mysqlrouter/mysql.sock' (2)                                                           

The above exception was the direct cause of the following exception:                                                                                                                                               

Traceback (most recent call last):                                                                                                                                                                                 
  File "/var/lib/juju/agents/unit-mysql-test-app-0/charm/./src/charm.py", line 210, in _stop_continuous_writes                                                                                                     
    last_written_value = self._max_written_value()                                                                                                                                                                 
  File "/var/lib/juju/agents/unit-mysql-test-app-0/charm/./src/charm.py", line 221, in _max_written_value                                                                                                          
    with MySQLConnector(self._database_config) as cursor:                                                                                                                                                          
  File "/var/lib/juju/agents/unit-mysql-test-app-0/charm/src/connector.py", line 42, in __enter__                                                                                                                  
    self.connection = mysql.connector.connect(**self.config)                                                                                                                                                       
  File "/var/lib/juju/agents/unit-mysql-test-app-0/charm/venv/mysql/connector/pooling.py", line 293, in connect                                                                                                    
    return CMySQLConnection(*args, **kwargs)                                                                                                                                                                       
  File "/var/lib/juju/agents/unit-mysql-test-app-0/charm/venv/mysql/connector/connection_cext.py", line 120, in __init__
    self.connect(**kwargs)                                                                               
  File "/var/lib/juju/agents/unit-mysql-test-app-0/charm/venv/mysql/connector/abstracts.py", line 1181, in connect
    self._open_connection()                                                                              
  File "/var/lib/juju/agents/unit-mysql-test-app-0/charm/venv/mysql/connector/connection_cext.py", line 296, in _open_connection
    raise get_mysql_exception(                                                                           
mysql.connector.errors.InterfaceError: 2002 (HY000): Can't connect to local MySQL server through socket '/var/snap/charmed-mysql/common/run/mysqlrouter/mysql.sock' (2)

The above exception was the direct cause of the following exception:                                                                                                                                               

Traceback (most recent call last):                                                                       
  File "/var/lib/juju/agents/unit-mysql-test-app-0/charm/./src/charm.py", line 208, in _stop_continuous_writes
    for attempt in Retrying(stop=stop_after_delay(60), wait=wait_fixed(5)):                                                                                                                                        
  File "/var/lib/juju/agents/unit-mysql-test-app-0/charm/venv/tenacity/__init__.py", line 384, in __iter__                                                                                                         
    do = self.iter(retry_state=retry_state)                                                              
  File "/var/lib/juju/agents/unit-mysql-test-app-0/charm/venv/tenacity/__init__.py", line 363, in iter                                                                                                             
    raise retry_exc from fut.exception()                                                                 
tenacity.RetryError: RetryError[<Future at 0x7efc336c6410 state=finished raised InterfaceError>]                                                                                                                   
unit-mysql-test-app-0: 10:41:43 INFO unit.mysql-test-app/0.juju-log database:6: Stop continuous writes                                                                                                             
unit-mysql-test-app-0: 10:41:43 INFO juju.worker.uniter.operation ran "database-relation-broken" hook (via hook dispatching script: dispatch)
unit-mysql-router-0: 10:41:44 INFO juju.worker.uniter.operation ran "cos-agent-relation-departed" hook (via hook dispatching script: dispatch)
unit-mysql-test-app-0: 10:41:44 INFO juju.worker.uniter.operation ran "application-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-mysql-router-0: 10:41:45 ERROR unit.mysql-router/0.juju-log database:6: Failed to run MySQL Shell script:
import json                                         
import mysqlsh                                      
import traceback                                    

try:                                                
    shell.connect("relation-7:***@10.205.193.79:3306")                                                                                                                                                             
    import json                                     

    cluster_set = dba.get_cluster_set()                                                                  
    routers = cluster_set.list_routers()                                                                 
    # mysqlsh objects are weird—they quack (i.e. duck typing) like standard Python objects (e.g. list,                                                                                                             
    # dict), but do not serialize to JSON correctly.                                                     
    # Cast to str & load from JSON str before serializing                                                                                                                                                          
    routers = json.loads(str(routers))                                                                   
    with open("/tmp/mysqlsh_output.json", "w") as file:                                                                                                                                                            
        json.dump(routers, file)                                                                         
except mysqlsh.DBError as exception:                                                                     
    error = {                                       
        "message": str(exception),                                                                       
        "code": exception.code,                                                                          
        "traceback_message": "".join(traceback.format_exception(exception)),                                                                                                                                       
    }                                               
else:                                               
    error = None                                    
with open("/tmp/mysqlsh_error.json", "w") as file:                                                       
    json.dump(error, file)                                                                               

MySQL client error 1045
MySQL Shell traceback:
Traceback (most recent call last):
  File "<string>", line 6, in <module>
mysqlsh.DBError: MySQL Error (1045): Shell.connect: Access denied for user 'relation-7'@'juju-5ca5f0-1.lxd' (using password: YES)

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/src/mysql_shell/__init__.py", line 105, in _run_code
    raise ShellDBError(**exception)
mysql_shell.ShellDBError: MySQL Error (1045): Shell.connect: Access denied for user 'relation-7'@'juju-5ca5f0-1.lxd' (using password: YES)
unit-mysql-router-0: 10:42:03 ERROR unit.mysql-router/0.juju-log database:6: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/./src/machine_charm.py", line 124, in <module>
    ops.main.main(MachineSubordinateRouterCharm)
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/venv/ops/main.py", line 441, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/venv/ops/framework.py", line 344, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/venv/ops/framework.py", line 841, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/venv/ops/framework.py", line 930, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/src/abstract_charm.py", line 295, in reconcile
    if not workload_.status or workload_.status == ops.WaitingStatus():
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/src/workload.py", line 360, in status
    if not self.shell.is_router_in_cluster_set(self._router_id):
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/src/mysql_shell/__init__.py", line 221, in is_router_in_cluster_set
    self._run_code(
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/src/mysql_shell/__init__.py", line 105, in _run_code
    raise ShellDBError(**exception)
mysql_shell.ShellDBError: MySQL Error (1045): Shell.connect: Access denied for user 'relation-7'@'juju-5ca5f0-1.lxd' (using password: YES)
unit-mysql-router-0: 10:42:03 ERROR juju.worker.uniter.operation hook "database-relation-broken" (via hook dispatching script: dispatch) failed: exit status 1

Additional context

github-actions[bot] commented 5 months ago

https://warthogs.atlassian.net/browse/DPE-3827

carlcsaposs-canonical commented 5 months ago

guess: issue with secrets or on server end

taurus-forever commented 5 months ago

@shayancanonical I believe the problem in very outdated mysql in 8.0/stable which doesn't have secrets implemented at all. Should MySQL Router error here? Probably not... but this can be low priority if router works well with 8.0/beta which we are preparing as a next stable release soonish.

It is reproducible on server 8.0/beta + router dpe/edge?

taurus-forever commented 1 week ago

@shayancanonical I have executed your steps-to-reproduce and it looks like it is no longer topical:

image

Removing mysql-router as described (in the setup above), produces no errors in juju debug-log. Resolved as no longer topical, feel free to reopen if you notice it again. Tnx!