Closed Ingolifs closed 3 years ago
Thanks for the reproducible boom.
There is currently unfortunately a hard limit built into the C++ side of words having maximum length of 100. This can be seen here: https://github.com/bnosac/doc2vec/blob/master/src/doc2vec/common_define.h#L11. Another hard limit set at the C++ side is that the maximum number of words a document can have is 1000 (https://github.com/bnosac/doc2vec/blob/master/src/doc2vec/common_define.h#L14)
The boom probably occurs at https://github.com/bnosac/doc2vec/blob/3e947562a0a69e11eb292283116a4fdc9cf5c0f4/src/doc2vec/TaggedBrownCorpus.cpp#L88 or https://github.com/bnosac/doc2vec/blob/3e947562a0a69e11eb292283116a4fdc9cf5c0f4/src/doc2vec/TaggedBrownCorpus.cpp#L90
The package only allows to have words with a maximum length of 100 characters. The crash is caused by strcpy copying a larger amount into a memory block which only allows 100 characters, namely at https://github.com/bnosac/doc2vec/blob/master/src/rcpp_doc2vec.cpp#L147 https://github.com/bnosac/doc2vec/blob/master/src/rcpp_doc2vec.cpp#L254 I'll explicitely truncate the strings to 100 characters to avoid the crash at the C++ side but keep in mind your words should be smaller than 100 characters in size.
Fix using substr in commit https://github.com/bnosac/doc2vec/commit/7f584f9f0458691595c511480bcb940ebbc1fd93
It took me a little while to hunt down the cause of this crash...
It does this on my machine at the very least. This is on R 3.6.3.