skit-ai / dialogy-template-simple-transformers

Dialogy template using simple transformers.
MIT License
5 stars 3 forks source link

[Bug] uses intent column as labels for training and not tag column #62

Closed biswaroop1547 closed 2 years ago

biswaroop1547 commented 2 years ago

currently the template uses intent column as labels for training and not tag column whereas intent column actually now model predictions for new charon's dataframe output format.

more info

biswaroop1547 commented 2 years ago

63 this PR fixes it

biswaroop1547 commented 2 years ago

To convert a previously used csv into the new charon format use this snippet by putting the whole csv file path in CSV_PATH variable:

import pandas as pd
import json
import tqdm

CSV_PATH = ""

def new_charon_format_conversion(CSV_PATH):
    df = pd.read_csv(CSV_PATH)
    df.drop(columns=['tag'], inplace = True)
    df['tag'] = df['intent'].values

    for i, row in tqdm.tqdm(df.iterrows(), total = len(df)):
        try:
            intent_pred = json.loads(row['prediction'])['name']
        except:
            intent_pred = None

        df.iloc[i, df.columns.get_loc('intent')] = intent_pred

    df.to_csv(CSV_PATH[:-4] + 'new.csv', index = False)
    print(f"{CSV_PATH} saved!")

new_charon_format_conversion(CSV_PATH)