NLP Features EDA

[ijyliu]

• [ ] Sentiment - underlying counts histogram, sentiment score histogram, sentiment score by rating
• [ ] Tone - dictionary words counts histogram, 2d scatterplot (reduce dimension further if necessary) - coloring points with credit rating
• [ ] Numeric transparency - underlying counts sum stats and histogram, ratio histogram, everything overall and by credit rating
• [ ] Engagement - count of number of question marks sum stats and histogram, everything overall and by credit rating
• [ ] Readability - sum stats and histogram, everything overall and by credit rating
• [ ] FinBERT embeddings - use PCA (with Whiten=True) or tSNE to get to two dimensions, and then create a scatterplot with each class colored
• [ ] Be on the lookout for any outliers and investigate them (potentially drop or Winsorize)

Correlations

• [x] correlation of sentiment and tone
• [x] correlation of numeric transparency and readability

[ijyliu]

Reminder to update to using new dataset

# Limit to items in the finalized dataset
# list of files in '../../../Data/All_Data/All_Data_with_NLP_Features'
import os
file_list = [f for f in os.listdir(r'../../../Data/All_Data/All_Data_with_NLP_Features') if f.endswith('.parquet')]
# read in all parquet files
df = pd.concat([pd.read_parquet(r'../../../Data/All_Data/All_Data_with_NLP_Features/' + f) for f in file_list])

[ijyliu]

@seanzhou1207 to clean up plots and save to Output folder

[ijyliu]

(and update to new dataset)

[ijyliu]

current level of eda is good, though it will need to be rerun in future