EDA on All Data (merged data file)

[ijyliu]

Build on the work in: https://github.com/current12/Stat-222-Project/blob/main/Code/Exploratory%20Data%20Analysis/All%20Data%20EDA.ipynb

Items from #7, #8, #9

• [x] What is the distribution of call dates? Check we have good coverage from 2010-2016.
• [x] What is the distribution of companies and industry sectors?
• [x] What companies drop out and why?
• [x] Distribution of Rating and Next Rating
• [x] Summary statistics of all numeric variables
• [x] see if tabular financial data variables sum stats correspond with those of the Graph NN paper

NLP EDA

• [x] How many tokens is the average call? (nltk tokenizer?)
• [x] Other existing NLP EDA items
• [x] (If time allows) Run Named Entity Recognition (spacy package) to get the number of companies mentioned in a call

Additional

• [x] write simple description of data cleaning

[ijyliu]

send pdf of notebook run on full data

and notebook code of eda

and sample dataset

write simple description of data cleaning

cc TA Zhexiao

[ijyliu]

@current12 actually, can you make a separate notebook that does the NLP EDA on the transcript features? it's very slow to run them in the main notebook. we can just add it as an optional thing if they want to look at them.

Here's the code to load the dataset. You can focuse on the Transcript column.

# Load in parquet file
# ~\Box\STAT 222 Capstone\Intermediate Data\all_data.parquet
df = pd.read_parquet(r'~\Box\STAT 222 Capstone\Intermediate Data\all_data.parquet')

[ijyliu]

@current12

Actually I'm finished with my part of

https://github.com/current12/Stat-222-Project/blob/main/Code/Exploratory%20Data%20Analysis/All%20Data%20EDA.ipynb

so if you want to do NLP and other stuff at the bottom of the same notebook, that's fine.

I left you the NLP stuff and also checking the tabular financial statement variables more carefully since you are most familiar with those variables. there's also code setup at the top of the notebook to switch the code over to using a sample dataset - since the actual data is 200MB+ and that's too much to send, we can submit with all_data_sample.csv, which is 100 firms.

[ijyliu]

if it's not too hard/doesn't take long to re-run, i'd comment out the financial statements unit correction in this EDA file so that they can see the true values (even if they're weird-looking)

also, can you do transcript length in words, not characters? (or both, but words is more meaningful) this applies to both the mean calculation and the distribution plot

otherwise, looks good to me! make sure the final pdf we turn in is run on the full data.

[ijyliu]

for number of words, I suggest just the standard nltk tokenizer, then strip punctuation tokens and count

[current12]

for number of words, I suggest just the standard nltk tokenizer, then strip punctuation tokens and count

I have uploaded the version with number of words. But I didn't comment out the financial statements unit correction.

[ijyliu]

if you're not going to comment it out, can you at least show the summary statistics before and after so there's a record of the original data?

[ijyliu]

i'm also a little concerned because it looks like, for example, we obliterated all observations in 2010. so our year plot is inaccurate if we don't comment it out

[ijyliu]

you should fix it to not affect these

and only the other variables

[current12]

I think it's not due to the financial statements unit correction. You can see the full dataset pdf. there are 2010

I think it's due to the lack in the original sample data

[ijyliu]

ok. still suggest adding summary stats before you correct it though

[current12]

np

[ijyliu]

nice job, once that's done i'd say good to submit

anyone else with code comments/who want's to review, speak quickly

[current12]

and I removed the correction part

[current12]

I have uploaded the latest eda notebook and pdf.

[ijyliu]

@current12 i'm going to continue working on the remaining things here. feel free to work on the other issues you're on

[ijyliu]

split company dropout into a new issue #29