Closed nantonop closed 4 years ago
Hello Nick, thank you. I do not think it is a good idea. I use file mapping into memory for fast parsing/tokenization. In case of many files it will be much slower. You can write a simple shell script to merge files from subdirectories to one large file.
Hi Max,
Hmm, good point.
I'm asking this because my training data is about 400GB. I can merge all the text files but will there be any issues e.g. memory issues by processing such a huge text file?
Thanks,
Nick
Hi Nick, I've never tried to mmap so large files. But it is possible to mmap files larger than both the physical memory and swap space. In my experience I trained a w2v model on 60GB data set with 32GB RAM. By the way, whole Wikipedia (English texts) is about 20GB only...
Hi Max,
Your word2vec implementation compiles and works fine - is also fast. Nice work!
Will it be possible, instead of pointing to a single train text corpus file option (-f [file] or --train-file [file]) to also point to a directory containing many text files and use those text files in that directory as training corpus? (i.e. something like --train-directory ~/Desktop/trainningData)
Thanks!