samba-in-kubernetes / sambacc

A library and CLI tools for managing Samba servers in containers.
GNU General Public License v3.0
13 stars 16 forks source link

sambacc: avoid logging an error if cluster is being torn down #131

Closed phlogistonjohn closed 2 months ago

phlogistonjohn commented 3 months ago

Saw this in a ceph teuthology run:

  2024-08-20 20:39:57,289: DEBUG: Creating RADOS connection
 2024-08-20 20:39:57,333: INFO: cluster meta content changed
 2024-08-20 20:39:57,333: DEBUG: cluster meta: previous={'nodes':
[{'pnn': 0, 'identity': 'smb.adctdb1.0.0.ceph0.kdlxgn', 'node':
'192.168.76.200', 'state': 'ready'}, {'pnn': 1, 'identity':
'smb.adctdb1.1.0.ceph1.ngbqkk', 'node': '192.168.76.201', 'state':
'ready'}, {'pnn': 2, 'identity': 'smb.adctdb1.2.0.ceph2.rhmqnu', 'node':
'192.168.76.202', 'state': 'ready'}], '_source': 'cephadm'} current={}
 2024-08-20 20:39:57,333: ERROR: error during ctdb_monitor_nodes: max()
arg is an empty sequence, count=0
 Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/sambacc/commands/ctdb.py", line
479, in catch
    yield
  File "/usr/lib/python3.9/site-packages/sambacc/commands/ctdb.py", line
360, in ctdb_monitor_nodes
    ctdb.monitor_cluster_meta_changes(
  File "/usr/lib/python3.9/site-packages/sambacc/ctdb.py", line 561, in
monitor_cluster_meta_changes
    expected_nodes = _cluster_meta_to_ctdb_nodes(
  File "/usr/lib/python3.9/site-packages/sambacc/ctdb.py", line 506, in
_cluster_meta_to_ctdb_nodes
    pnn_max = max(n["pnn"] for n in nodes) + 1  # pnn is zero indexed
 ValueError: max() arg is an empty sequence

I could see from the ceph logs the smb cluster was being removed right around this time. If we had nodes and they suddenly vanish we're likely in the process of getting removed and we raced a tad with cephadm removing services while the smb mgr module was removing the contents of the .smb pool.

phlogistonjohn commented 2 months ago

@mergifyio rebase

mergify[bot] commented 2 months ago

rebase

✅ Branch has been successfully rebased