Model risk management

How to build models that move quickly through validation and audit: With regulators and policymakers seriously addressing the challenges of AI in finance and banks starting to demand more measurable profits around the use of data, data practices are forced to step up their game in the delivery of ML if they want to drive competitive insights reliable enough for business to trust and act upon. This utility library will automate the generation of PDF report for submitted new models as part of a model risk management practice. Using mlflow, delta lake, unity catalog and their respective APIs, we aim at bringing both the technical and business context surrounding your model submission, reducing time to market and facilitating independent validation of financial and non-financial models.

antoine.amend@databricks.com

Usage

from databricks.mrm import ModelRiskApi

mrm = ModelRiskApi(
    databricks_workspace_url,
    databricks_token
)

mrm.generate_doc(
    model_name=model_name, # name of the model on ML registry
    model_version=model_version, # version of the model (optional, default is latest)
    output_file=output_file # name of the output file for PDF document
)

See example output for a given test model done using an existing solution accelerator.

Make sure to have both wkhtmltopdf and graphviz binary installed

sudo apt-get install -y graphviz wkhtmltopdf

Command line

Should you need to run the same from a command line utility, please refer to databricks.py.

pip install -r requirements.txt
python databricks.py \
    --db-workspace my-workspace-url \
    --db-token my-workspace-token \
    --model-name my-model-name \
    --model-version my-model-version \
    --output my-model-output.pdf

Template documentation

This utility library provides a technical foundation and framework to automatically generate the document and represents one side of the coin. The other side relates to the design, the structure and the content of the document that relates to the model in scope. With subject matter expertise supporting several model frameworks, regulator guidelines and custom needs of many FS institutions, the MRM suite from EY provides proven model documentation templates corresponding to business domains in the financial services industry.

See an example of template document for Credit Risk Adjudication use case. Available as a Ipython notebook file, hence easily accessible through Databricks environment, this template provides the necessary placemat through markdown comments to generate the documentation required for Model Risk Management.

License

© 2023 Databricks, Inc. All rights reserved. The source is provided subject to the Databricks License [https://databricks.com/db-license-source]. All included or referenced third party libraries are subject to the licenses set forth below.