xapi-project / xcp-idl

Interface descriptions for services running on an XCP host.
Other
9 stars 41 forks source link

Fix backtraces in xenopsd-xc #315

Closed edwintorok closed 4 years ago

edwintorok commented 4 years ago

A failing HOST.stat call logged just the exception, but not the backtrace:

Sep  1 10:55:07 localhost xenopsd-xc: [error||72 ||backtrace] dbsync (update_env) R:a771d46f95c9 failed with exception Xenctrlext.Unix_error(57, "105: No buffer space availa
ble")
Sep  1 10:55:07 localhost xenopsd-xc: [error||72 ||backtrace] Raised Xenctrlext.Unix_error(57, "105: No buffer space available")
Sep  1 10:55:07 localhost xenopsd-xc: [error||72 ||backtrace] 1/1 xenopsd-xc Raised at file (Thread 72 has no backtrace table. Was with_backtraces called?, line 0
Sep  1 10:55:07 localhost xenopsd-xc: [error||72 ||backtrace]

Although with_backtraces was definitely called by Debug.with_thread_associated:

    let stat _ dbg =
      Debug.with_thread_associated dbg
        (fun () ->
          debug "HOST.stat" ;
          let module B = (val get_backend () : S) in
          B.HOST.stat ())
        ()

This is because with_backtraces removes the thread-local backtrace table that it has created on exit and returns an exception together with its backtrace. The backtrace was getting ignored, and re-queried in Debug.log_backtraceexn. However that only works if Debug.with* calls are nested.

Fix this so we always use the backtrace if available instead of querying it.

log_backtrace can also be called externally, so make sure we still attempt to retrieve the backtrace from the exception if the backtrace is empty.

After the fix we now get a proper backtrace:

Sep  1 13:47:07 localhost xenopsd-xc: [error||44 ||backtrace] dbsync (update_env) R:531918011f4b failed with exception Xenctrlext.Unix_error(57, "105: No buffer space availa
ble")
Sep  1 13:47:07 localhost xenopsd-xc: [error||44 ||backtrace] Raised Xenctrlext.Unix_error(57, "105: No buffer space available")
Sep  1 13:47:07 localhost xenopsd-xc: [error||44 ||backtrace] 1/3 xenopsd-xc Raised at file xenopsd/xc/xenops_server_xen.ml, line 834
Sep  1 13:47:07 localhost xenopsd-xc: [error||44 ||backtrace] 2/3 xenopsd-xc Called from file xenopsd/xc/xenops_server_xen.ml, line 930
Sep  1 13:47:07 localhost xenopsd-xc: [error||44 ||backtrace] 3/3 xenopsd-xc Called from file xapi-idl/lib/debug.ml, line 220
Sep  1 13:47:07 localhost xenopsd-xc: [error||44 ||backtrace]
Sep  1 13:47:07 localhost xenopsd-xc: [error||44 ||xenops_interface] Xenctrlext.Unix_error(57, "105: No buffer space available") (File "xapi-idl/xen/xenops_interface.ml", li
ne 168, characters 51-58)
Sep  1 13:47:07 localhost xapi: [error||0 |dbsync (update_env) R:531918011f4b|xenops_interface] Xenops_interface.Xenopsd_error([S(Internal_error);S(Xenctrlext.Unix_error(57,
 "105: No buffer space available"))]) (File "xen/xenops_interface.ml", line 160, characters 49-56)

The is_important marking is needed inside with_backtraces, otherwise the backtrace wouldn't be stored, and we would only get the fallback backtrace on Backtrace.empty.