Closed hermitdave closed 5 years ago
@hugolpz apologies for the delays.. i downloaded the tar.gz files on another machine and never got around to running it. Based on early user feedback, I implemented checks to ensure duplicates were reduced. The subtitle folders often contain multiple subtitles. I have taken to only picking up a single file if multiple files are found. Programmatically identifying language is harder.
I could do basic checks like this language should only have latin / cryllic character set.
I am going to use this project to do language detection https://github.com/TechnikEmpire/language-detection
Its a .NET port of https://code.google.com/archive/p/language-detection/
Hello Dave, cool to see you back, Thanks for considering my input.
I could do basic checks like this language should only have latin / cyrillic character set. I think you "latin/cyrillic" would mean russian and similar, as subtitles often contains latin words for all languages, at least for "ok" and other basic ones.
As for :
I am going to use this project to do language detection
Priority to get job done. Then, use more popular languages (python, js, java) so community can jump in if want.
Or others :
packaged with support for 82, 188, or 402 languages
Thanks @hugolpz I have reworked the code and added language lookup using .NET for now. Its slowly chugging along generating the files as we speak. I am downloading the dataset again - they changed from tar.gz to zip so wanted to ensure i had the latest set..
I will start uploading soon
Ahahahahahahahahaha. I didn't expected that fast :+1:
I tried to create my own list and I bumped into lot of pollution. I share with you my findings: lot of English names and basic English in French subtitles.
Noise: Lot of characters names, a bunch of basic English words. I review and cleaned up the 6000 to 8000th list: out of 2000 items, I had to edit 206 modifications (10%) and make 82 deletions (4%) (diff, open view-source:https://lingualibre.fr/index.php?title=List%3AFra%2Fsubtlex-for-user-Penegal-06001-to-08000&type=revision&diff=83866&oldid=83864 and search for "diff-deletedline" and "diff-addedline").
On my side, I'am building a list of "words to preprocess before calculating stats". Some are to delete (Individual names). Some are to edit, to move all to lowercase or to capitalized. This kind of thing?
@hugolpz i am at home (case of mild flu) which means I can do other things.
That's pretty neat 👍 I force lower casing to ensure small reduction in noise
Maybe I should create another output with the words / characters that were filtered out ?
Yes, forcing lower case is smart. For names I had to inject back the Capitalization manually. But I'am not sure it really worth it has my end goal is to record words : my recording can be all lower case as well.
See http://opus.nlpl.eu/OpenSubtitles2018.php :smile: To have it under hands ! :smiley:
Method:
I found out Subtlex-pl (2014) discussion interesting. I cross checked below with Lison & Tiedemann (2016), they barely talk about duplicata. So maybe Lison & Tiedemann did the anti-duplicatia work --very likely given the extensive processing on the data--.
Subtlex-pl (2014) : article, non-free. The section on data clean up is interesting. Some part are at reach. Other not. 3 methods are mentioned :
P. Lison and J. Tiedemann (2016), "OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles", In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016) http://stp.lingfil.uu.se/~joerg/paper/opensubs2016.pdf .
Also related to : #2