Open tybrs opened 1 year ago
singhniraj08
commented
1 year ago @tybrs,
In official documentation, it shows that multi-class classification metrics is supported by TFMA. Also, MultiClassConfusionMatrixPlot is supported by TFMA, please refer here.
Thank you!
tybrs
commented
1 year ago @singhniraj08 thanks for getting back!
I do see that MultiClassConfusionMatrixPlot is supported, which we use regularly on TF models. The question is whether MultiClassConfusionMatrixPlot is supported with DataFrames using analyze_raw_data.
If it is supported: (1) how should the DataFrame be structured to support multiple prediction classes (2) how should the EvalConfig look
embr
commented
1 year ago Thanks for bringing this to our attention. This looks like a bug in some code that is attempting to construct pyarrow.RecordBatches in a special way. I suspect this was intended to limit the supported types to avoid problems downstream, but it seems that it is too strict right now. It is possible that this may lead to more cryptic errors downstream, but at least for the toy DataFrame you've provided, I was able to get things working by just directly calling pyarrow.RecordBatch.from_pandas which correctly handles the nested array type:
ifrom google.protobuf import text_format
import apache_beam as beam
import pandas as pd
import pyarrow as pa
import tensorflow_model_analysis as tfma
from tfx_bsl.arrow import table_util
df = pd.DataFrame({
'predictions': [[0.6, .7, .8], [0.4, 0.5, 0.6]],
'label': [1, 0]})
eval_config = text_format.Parse(
"""
model_specs {
label_key: 'label'
prediction_key: 'predictions'
}
metrics_specs {
metrics { class_name: "AUC" }
binarize { class_ids { values: [1,2] } }
}
""", tfma.EvalConfig())
output_path = 'raw-analysis-output'
with beam.Pipeline() as p:
_ = (
p
| 'CreateRecordBatches' >> beam.Create([
table_util.CanonicalizeRecordBatch(
pa.RecordBatch.from_pandas(df))])
| 'ExtractEvaluateAndWriteResults' >> tfma.ExtractEvaluateAndWriteResults(
eval_config=eval_config,
output_path=output_path))
for metrics_per_slice in tfma.load_metrics(output_path):
print(metrics_per_slice)I'll look into supporting this case more generically, but I figured I'd share this workaround now to unblock you.
System information
pip install tensorflow-model-analysisDescribe the problem
I am currently trying to get
tfma.analyze_raw_datato work withMultiClassConfusionMatrixPlotwhich has multiple prediction values per record. Is this not supported? I will be happy to provide any further details or run any further tests.Details
Currently
tfma.analyze_raw_datadoes not seem to work with metrics for multi classification tasks (e.g.tfma.metrics.MultiClassConfusionMatrixPlot). However, I do not see this limitation documented anywhere.The prediction column for a multi classification column will be a series of whose values are a list or array (e.g.,.
pd.DataFrame({'predictions': [[0.2, .3, .5]], 'label': [1]}))The
tfma.analyze_raw_datafunciton usestfx_bsl.arrow.DataFrameToRecordBatchto convert a Pandas DataFrame to Arrow RecordBatch. The problem, however, is that it encodes columns with the dtype ofobjectas apyarrow.Binary. Since a column that has lists or arrays as values has a dtype ofobject, these columns are being encoded as apyarrow.Binaryinstead of the relevant pyarrow list-like type.Source code / logs
Error
Temporary fix
If I change/patch
tfx_bsl.arrow.DataFrameToRecordBatchas follows, it seems to work, but I doubt this is a solution.