search

apache / datafusion

Apache DataFusion SQL Query Engine
https://datafusion.apache.org/
Apache License 2.0
6.06k stars 1.15k forks source link

Add Backtraces to DataFusionError #5283

Open tustvold opened 1 year ago

tustvold commented 1 year ago

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

It is fairly common to get errors from DataFusion such as in https://github.com/apache/arrow-ballista/issues/665

2023-02-12T22:26:10.297254Z ERROR tokio-runtime-worker ThreadId(02) ballista_scheduler::scheduler_server::query_stage_scheduler: Error planning job t9P5iXR: DataFusionError(SchemaError(FieldNotFound { field: Column { relation: Some("part"), name: "p_brand" }, valid_fields: [Column { relation: None, name: "COUNT(DISTINCT partsupp.ps_suppkey)" }] }))

Which do not make it easy to understand the context in which the error was encountered. Typically the approach I follow is to use an interactive debugger, such as GDB, to catch the error. However, this requires having a local reproducer, which isn't always possible.

Describe the solution you'd like

Backtraces are a commonly used mechanism to automatically provide context on where an error was encountered. Whilst they have issues, especially in async contexts, they are better than the current situation where diagnosing such issues is non-trivial.

Adding a backtrace to the error variants that have corresponding From conversions, i.e. ArrowError, ParquetError, AvroError, ObjectStore, IoError, SchemaError, etc... would automatically capture the context in which this conversion is performed.

APIs can then use DataFusionError::find_root to get the leaf variant, and from that extract the backtrace if any.

This is predicated on the fact that DataFusionError is rarely used to implement control flow, and so the cost of obtaining a backtrace is only paid on exceptional paths.

Describe alternatives you've considered

Additional context Add any other context or screenshots about the feature request here.

comphead commented 1 year ago

This is great and really needed. I was struggling several times to navigate to the rootcause from DF to arrow-rs. I believe doing the same from Ballista is even more complicated.

Btw we can try add a function name to the caller like https://stackoverflow.com/a/40234666/11285252 whats happening now is rethrow the DataFusionError multiple times so the error origin get lost completely

Similar behaviuor can be found https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=c876fabb792c97ff973ce193831a98de

tustvold commented 1 year ago

I think we can use #[track_caller] on the From implementations so that the root of the stack will be the function name

https://github.com/rust-lang/rfcs/blob/master/text/2091-inline-semantic.md