opendp / opendp

The core library of differential privacy algorithms powering the OpenDP Project.
https://opendp.org
MIT License
277 stars 46 forks source link

Usability: Python Error Framework #1420

Open chikeabuah opened 1 month ago

chikeabuah commented 1 month ago
Screen Shot 2024-03-21 at 9 44 45 AM

Consider restructuring Python error framework for Python users.

From study "Evaluating the Usability of Differential Privacy Tools with Data Practitioners" by Ngong et al.

chikeabuah commented 1 month ago
Screen Shot 2024-03-21 at 9 48 19 AM
mccalluc commented 1 month ago

Just considering the Rust internals appearing in python stacks, I think this has improved:

Shoeboxam commented 1 month ago

There are many error messages coming up from Rust, but do not require any knowledge of Rust. For this reason, many errors should still bubble up from Rust. For instance, errors like: "scale must not be negative" or "candidates must be increasing" (on a quantile scoring transformation) are meaningful and actionable, even though they are Rust errors.

I think the main action point here is hiding the Rust stack traces, which is the default behavior now (thanks for the links @mccalluc!)

Shoeboxam commented 1 month ago

Our API design has also adjusted to take the previous transformation's output domain and output metric as arguments to the next constructor, which I think results in fewer domain or metric mismatch errors. Where before you'd get a domain mismatch error, now the library should state the expected domain/metric.

Shoeboxam commented 1 month ago

I'm of course open-minded to other suggestions. On the Python side, we could have separate exception classes for each error variant in Rust. Right now this error variant is just interpolated into the error message. We could also update the error variants used inside Rust.

It would probably also be worthwhile to do a pass over each error constructed in rust (either via fallible! or err!) to ensure that they are actionable/meaningful to users in bindings languages.