ceph / ceph-medic

find common issues in ceph clusters
MIT License
22 stars 18 forks source link

ensure (properly decoded) strings are returned on exceptions #111

Closed alfredodeza closed 5 years ago

alfredodeza commented 5 years ago

When exceptions occur, a helper grabs all the information and returns it. If during the process any non-serializable object is added, the whole process crashes as the local instance can't do anything about the object.

In the case below, this happened because the tmp directory was being captured, containing a binary monmap which caused a UnicodeError that had the undecodable contents of the monmap. The fix has to go through the values in remote/functions.py in the capture_exception helper, and ensure decoded values are set, as strings

  File "/home/alfredo/.virtualenvs/ceph-medic/lib/python3.6/site-packages/remoto-1.1.2-py3.6.egg/remoto/backends/__init__.py", line 160, in wrapper
    return self.channel.receive()
  File "/home/alfredo/.virtualenvs/ceph-medic/lib/python3.6/site-packages/execnet-1.6.0-py3.6.egg/execnet/gateway_base.py", line 728, in receive
    raise self._getremoteerror() or EOFError()
execnet.gateway_base.RemoteError: Traceback (most recent call last):
  File "<string>", line 1374, in _save
KeyError: <class 'builtin_function_or_method'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1063, in executetask
  File "<string>", line 1, in do_exec
  File "<remote exec>", line 155, in <module>
  File "<string>", line 708, in send
  File "<string>", line 1345, in dumps_internal
  File "<string>", line 1363, in save
  File "<string>", line 1381, in _save
  File "<string>", line 1462, in save_dict
  File "<string>", line 1456, in _write_setitem
  File "<string>", line 1381, in _save
  File "<string>", line 1462, in save_dict
  File "<string>", line 1456, in _write_setitem
  File "<string>", line 1381, in _save
  File "<string>", line 1462, in save_dict
  File "<string>", line 1456, in _write_setitem
  File "<string>", line 1379, in _save
DumpError: can't serialize <class 'builtin_function_or_method'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/alfredo/.virtualenvs/ceph-medic/bin/ceph-medic", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/alfredo/python/ceph-medic/bin/ceph-medic", line 6, in <module>
    main.Medic()
  File "/home/alfredo/python/ceph-medic/ceph_medic/main.py", line 45, in __init__
    self.main(argv)
  File "/home/alfredo/python/ceph-medic/ceph_medic/decorators.py", line 46, in newfunc
    return f(*a, **kw)
  File "/home/alfredo/python/ceph-medic/ceph_medic/main.py", line 145, in main
    parser.dispatch()
  File "/home/alfredo/.virtualenvs/ceph-medic/lib/python3.6/site-packages/tambo-0.4.0-py3.6.egg/tambo/dispatcher.py", line 21, in dispatch
    result = instance.main()
  File "/home/alfredo/python/ceph-medic/ceph_medic/check.py", line 65, in main
    collector.collect()
  File "/home/alfredo/python/ceph-medic/ceph_medic/collector.py", line 194, in collect
    metadata[node_type][hostname] = get_node_metadata(conn, hostname, cluster_nodes)
  File "/home/alfredo/python/ceph-medic/ceph_medic/collector.py", line 124, in get_node_metadata
    node_metadata['paths'] = collect_paths(conn)
  File "/home/alfredo/python/ceph-medic/ceph_medic/collector.py", line 88, in collect_paths
    path_metadata[p] = get_path_metadata(conn, p, **kw)
  File "/home/alfredo/python/ceph-medic/ceph_medic/collector.py", line 105, in get_path_metadata
    files[i] = conn.remote_module.stat_path(i, None, None, kw.get('get_contents'))
  File "/home/alfredo/.virtualenvs/ceph-medic/lib/python3.6/site-packages/remoto-1.1.2-py3.6.egg/remoto/backends/__init__.py", line 170, in wrapper
    raise RuntimeError(exc_line)
RuntimeError: DumpError: can't serialize <class 'builtin_function_or_method'>