[SIIT] Clean 3G pantip Data

[ArthurMinovsky]

Description

Analyze Pantip 3G dataset and see if there are any

• formatting error
• Comments that are duplicated or near-duplicated with others in the same post
• Comments or Post that is spam, SEO, or hate speech
• Tags that should not be included, should be upsampling because of high/low quality comments has tags)
• Do Topic modeling on the dataset to know domain of the datasets
• Others that you find important to fully understand Pantip 3G datasets

Ref Near Duplicate:

• https://www.techtarget.com/whatis/definition/fuzzy-search#:~:text=A%20fuzzy%20search%20is%20a,how%20a%20word%20is%20spelled.
• https://github.com/seatgeek/thefuzz

Outcomes:

• Jupyter notebook that do all