esherman-credo
commented
1 year ago Github won't let me upload a notebook as evidence and I don't want to push my testing notebook into the repo. This should be sufficient to replicate the bug (need to have the usual training_script.py in the right folder).
The output of the last two cells will differ. The first will be the "correct" results when omitting training data. The second reveals the bug with training data included. The second should return two sets of results: one for the training data and one for the assessment data (or possibly just the assessment data...).
%load_ext autoreload
%autoreload 2
from credoai.lens import Lens
from credoai.artifacts import TabularData, ClassificationModel
from credoai.evaluators import *
from credoai.governance import Governance
import numpy as np
%run ~/credoai_lens/docs/notebooks/training_script.py## Set up artifacts
credo_model = ClassificationModel(
'credit_default_classifier',
model
)
train_data = TabularData(
name = "UCI-credit-default-train",
X=X_train,
y=y_train,
sensitive_features=sensitive_features_train
)
test_data = TabularData(
name='UCI-credit-default-test',
X=X_test,
y=y_test,
sensitive_features=sensitive_features_test
)
# pipeline scan be specifed using a sklearn-like style
metrics = ['accuracy_score', 'roc_auc_score']
pipeline = [
(Performance(metrics), 'Performance Assessment'),
(ModelFairness(metrics), "Fairness Assessment"),
]# and added to lens in one fell swoop
lens = Lens(
model=credo_model,
assessment_data=test_data,
# training_data=train_data,
pipeline=pipeline
)
lens.run()
## Getting evidence or results out
results = lens.get_results()
results['Fairness Assessment']###RUN WITHOUT TRAINING DATA SPECIFIED
# pipeline scan be specifed using a sklearn-like style
metrics = ['accuracy_score', 'roc_auc_score']
pipeline = [
(Performance(metrics), 'Performance Assessment'),
(ModelFairness(metrics), "Fairness Assessment"),
]
# and added to lens in one fell swoop
lens = Lens(
model=credo_model,
assessment_data=test_data,
training_data=train_data,
pipeline=pipeline
)
lens.run()
results = lens.get_results()
results['Fairness Assessment']
IanAtCredo
Expected Behavior
Lens should return results for each specified dataset, and for each evaluator (e.g. 2 data sets + 1 evaluator --> 2 sets of results).
Actual Behavior
Lens is currently only returning one result set when both
assessment_dataandtraining_dataare passed to theLens()constructor. It appears that only the results from thetraining_dataare processed (likely because that parameter is listed 2nd?)Creating data objects as follows:
Creating pipeline and
Lensobject as followsExpected Results: dictionary with 2 sets each of Performance and ModelFairness results. Running as above yields results that look like this (not enough entries and entries are clearly on training_data since performance is so strong):
Commenting out the line that passes
train_datatoLens()constructor yields performance that looks like this: