Open zywind opened 2 years ago
Hi zywind, Looks like TF2 based models are incompatible with metrics_callback. Could you try using Fairness Indicators as a metric (https://www.tensorflow.org/responsible_ai/fairness_indicators/guide)? In your case:
metrics_specs=[
tfma.MetricsSpec(
metrics=[
tfma.MetricConfig(class_name='ExampleCount'),
tfma.MetricConfig(class_name='BinaryAccuracy'),
tfma.MetricConfig(class_name='FairnessIndicators', config='{"thresholds":[0.3,0.5,0.7]}'),
]
)
]
This syntax should work with both TFX and non-TFX examples you've pasted.
Also, could you point me to the guide you have been using for Fairness Indicators so that I could capture it there?
Thank you @kumarpiyush for the information. I tried your suggestion but I get a different error:
IndexError: arrays used as indices must be of integer (or boolean) type [while running 'ExtractEvaluateAndWriteResults/ExtractAndEvaluate/EvaluateMetricsAndPlots/ComputeMetricsAndPlots()/ComputePerSlice/ComputeUnsampledMetrics/CombinePerSliceKey/WindowIntoDiscarding']
This might be because I'm using an outdated TFMA (v0.26)? My company will update our TFMA soon so I will test this again later.
As for the guide, I was just following the standard guide here: https://www.tensorflow.org/tfx/guide/fairness_indicators#compute_fairness_metrics. It suggested using the add_metrics_callbacks parameter. For TFX, there is no guide. I just found the fairness_indicator_thresholds parameter in the Evaluator's API. Adding a guide for TFX's Evaluator would be very useful.
@zywind are you by any chance applying transformations to your label? I experienced a similar issue when transforming string labels to one-hot encoded array's. In your case, you are using the output from the exampleGen component rather than the transform component. If your labels are strings, then your error is probably generated by the following line of code, coming from the one_hot function in tensorflow_model_analysis/metrics/metric_util.py:
tensor = np.delete(np.eye(target.shape[-1] + 1)[tensor], -1, axis=-1)
To solve this, you should take the output of the transform component rather than the exampleGen component. Again, I don't know your full situation, but this might be a possible reason your pipeline is failing.
@DirkjanVerdoorn Thanks for the suggestion. The labels are integer types so that's really not the problem.
Just to follow up on this, now our environment is updated to TFMA 0.31 and I can confirm that the fairness_indicator_thresholds parameter in Evaluator still doesn't work, but @kumarpiyush's method worked. @kumarpiyush You may want to update the guide here: https://www.tensorflow.org/tfx/guide/fairness_indicators#compute_fairness_metrics
System information
eval_config = tfma.EvalConfig( model_specs=[ tfma.ModelSpec(label_key='toxicity', signature_name='serve_tfexample') ], metrics_specs=[ tfma.MetricsSpec( metrics=[ tfma.MetricConfig(class_name='ExampleCount'), tfma.MetricConfig(class_name='BinaryAccuracy') ] ) ], slicing_specs=[ tfma.SlicingSpec(), tfma.SlicingSpec(feature_keys=['race']), ] )
evaluator = Evaluator( examples=example_gen.outputs['examples'], schema=schema_gen.outputs['schema'], model=trainer.outputs['model'], fairness_indicator_thresholds=[0.3, 0.5, 0.7], eval_config=eval_config, )
context.run(evaluator) eval_result_uri = evaluator.outputs['evaluation'].get()[0].uri eval_result = tfma.load_eval_result(eval_result_uri)
from tensorflow_model_analysis.addons.fairness.view import widget_view widget_view.render_fairness_indicator(eval_result=eval_result)
tfma_eval_result_path = './bert/tfma_eval_result' import tensorflow_model_analysis.addons.fairness.post_export_metrics.fairness_indicators from google.protobuf import text_format
metrics_callbacks = [ tfma.post_export_metrics.fairness_indicators(thresholds=[0.3, 0.5, 0.7]), ]
eval_config = tfma.EvalConfig( model_specs=[ tfma.ModelSpec(label_key=LABEL) ], metrics_specs=[ tfma.MetricsSpec( metrics=[ tfma.MetricConfig(class_name='ExampleCount'), tfma.MetricConfig(class_name='BinaryAccuracy') ] ) ], slicing_specs=[
An empty slice spec means the overall slice, i.e. the whole dataset.
] )
eval_shared_model = tfma.default_eval_shared_model( eval_saved_model_path='./checkpoints', add_metrics_callbacks=metrics_callbacks, eval_config=eval_config)
eval_result = tfma.run_model_analysis( eval_config=eval_config, eval_shared_model=eval_shared_model, data_location=validate_tf_file, output_path=tfma_eval_result_path)
from tensorflow_model_analysis.addons.fairness.view import widget_view widget_view.render_fairness_indicator(eval_result=eval_result)