Create better utilities to visualize errors in LabelFunctions

[rjurney]

Is your feature request related to a problem? Please describe.

I want better tools to show what the errors are in my Snorkel LabelFunctions based on a LabelModel being trained against validation data.

Describe the solution you'd like

Right now I have this utility working for me. I would like the capability below integrated with the API and I am willing to do the work if you will accept it. Unlike snorkel-extraction this is not specific to NLP.

The code for my utility is here: https://gist.github.com/rjurney/33c5e5f5610565785905dfd77083a0d3

A full use case is here: https://github.com/rjurney/amazon_open_source/blob/master/3_Amazon_Open_Source_Analysis.ipynb

from snorkel.analysis import get_label_buckets

ABSTAIN   = -1
GENERAL   = 0
API       = 1
EDUCATION = 2
DATASET   = 3
names  = ['GENERAL', 'API', 'EDUCATION', 'DATASET']

# Trim the fields for figuring out problems
df_viz = df_test[['full_name', 'description', 'readme']]

# Display all errors for debugging purposes
pd.set_option('display.max_rows', len(df_viz.index))

def get_mistakes(df, labels, names):
    """Take DataFrame and pair of actual/predicted labels/names and return a DataFrame showing those records."""
    df_fn = df.iloc[buckets[labels]]
    df_fn['probability'] = probs_test[buckets[labels], 1]
    df_fn['true label'] = names[0]
    df_fn['predicted label'] = names[1]
    return df_fn

def mistakes_df(df, label_model, L_test, y_test, names):
    """Compute a DataFrame of all the mistakes we've seen."""
    out_dfs = []

    probs_test = label_model.predict_proba(L=L_test)
    probs_test = probs_test >= 0.5

    buckets = get_label_buckets(y_test, probs_test[:, 1])

    for (actual, predicted) in buckets.keys():

        if actual != predicted:

            actual_name    = names[actual]
            predicted_name = names[predicted]

            out_dfs.append(
                get_mistakes(
                    df,
                    labels=(actual, predicted),
                    names=(actual_name, predicted_name)
                )
            )

    return out_dfs[0].append(
        out_dfs[1:]
    )

mistakes_df(df_viz, majority_model, L_test, y_test, names).head(202)

This shows:

Describe alternatives you've considered

snorkel-extraction does similar stuff but not this exactly and it is too ugly to use at the moment.

Additional context

I like apples.

[rjurney]

https://gist.github.com/rjurney/060927d0c59037579f6212bd11553133

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.