Closed lakeofsoft closed 4 years ago
Thanks @lakeofsoft let me try to look into it in the next few days
I re-uploaded the data files, if it doesn't work you can now download the unzipped text file as well: https://github.com/kavgan/nlp-text-mining-working-examples/tree/master/word2vec
Sorry, but I can't see any difference. Both text files (the one inside .gz and unpacked "reviews_data.txt") still contains binary data which appears to be a RAR archive.
Simply open the file in any text viewer/editor and search for "Rar!" string, or go directly to line #53936 (byte offset 0x036573B0).
As a side note, for anyone who (like me) is struggling to get the "reviews_data.txt" content (and not the mere lfs link), these commands might help:
> git lfs install
> git lfs pull
🤔 I wonder why it doesn't complain on my Mac, are you using a windows based machine?
ahh just saw that its windows.
yes, open(...) uses locale.getpreferredencoding(False) when encoding is not specified, and it is platform dependent. You can try it with:
open(...., encoding="mac_latin2") # will not crash (default for Mac)
or
open(...., encoding="cp1252") # will crash (default for Windows)
and it will read nothing on Android, where default encoding is "utf_8"
there is a binary RAR file snugged inside text lines of "reviews_data.txt.gz"
inside that RAR there is only one file named "usa_nevada_las-vegas_riviera_hotel_casino", which contains some duplicated lines from .gz file
this causes UnicodeDecodeError exception under Windows
you can use open(...., errors='replace') to replace binary data with ? marks