Data Exploration & Data Visualization // Pre vs Post text cleaning #PostDefense, #NiceToHave

[Wilsbert12]

I would like to show before / after effects of text cleaning:

• [ ] How much text did we cut? (total + percentage)
• [ ] How did the clean affect the description length? Do we still have so many outliers or did we eliminate them?

Graph per category type (primary category, sub category)

• [ ] Text length: Post, Post, Saved chars
• [ ] Miss descr.
• [ ] Dupl. descr.
• [ ] Number of errors

Since the question of cleaning html code etc. derives from some of the analysis of the text, I would suggest the following overall structure:

1. Setup / Data Exploration
2. Data Viz
3. Text Clean
4. Before / after text clean comparison

[Wilsbert12]

@PeterStieg @thomas-borer makes sense?

[PeterStieg]

@Wilsbert12, please note that currently we are not...

1. either cutting off lengthy description at same point via _[:charlimit]
2. or excluding outliers

[Wilsbert12]

@PeterStieg I know. But the outliers might be due to "junk" text (very long empty spaces, html code, repetitions etc.). I just thought if we are cleaning up the text, we might also show what we have archived.