AngelinaZhai
commented
1 year ago - Added additional dataframe processing to remove non-English language (using langdetect)
- Removed non-ascii (typical alphabetical) characters
- Removed around 1000 data points from the original 13.5k entries
- Implemented error detection and contingency for embed loading after data cleaning