search

fengkaicnic / word2vec

Automatically exported from code.google.com/p/word2vec
Apache License 2.0
0 stars 0 forks source link

The definition of sentences? #28

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago 
Hi, I am new to word2vec. I am preparing corpus in sentences using wikipedia 
dump. However the dump is pre-splitted in paragraphs which seems need to 
further be processed into sentences. 

My question is 

is it possible to directly train paragraphs instead of sentences? Or it is a 
must that word2vec (the SkipGram model) has to work with sentences. 

Since the algorithm trains the data by a context window, I didn't see much 
difference by add the extra window across sentences within the same paragraph.

Original issue reported on code.google.com by yel...@gmail.com on 24 Feb 2015 at 9:35

GoogleCodeExporter commented 8 years ago 
You can train with paragraphs instead of sentences, it should be OK.

Original comment by tmiko...@gmail.com on 25 Feb 2015 at 8:24