gojomo
commented
5 years ago To the extent it might be possible to double-check the randomization that's happening in cython, I ran the following cells in a notebook - which seem to indicate random_int32(&c.next_random) is not malfunctioning to be greater-than a sample_int value of 2**32 (the lowest value any of the never-downsampled, less-frequent words appear to have).
%load_ext Cython%%cython
from gensim.models.word2vec_inner cimport random_int32
import logging
cimport numpy as np
from numpy import random
RS = random.RandomState(1)
cdef unsigned long long next_random = (2**24) * RS.randint(0, 2**24) + RS.randint(0, 2**24)
cpdef unsigned long long cyrand():
return random_int32(&next_random)
cpdef unsigned long long show_nr():
return next_randomfor _ in range(1000000000):
if (cyrand() >= (2**32)):
print("whoops") # not seenThis somewhat decreases my confidence that the sample value is inherently involved: maybe it's just shrinking the corpus enough so that some other stochastic error becomes detectable. (For example, maybe the real bug is that the wrong vector/memory is often being updated; more epochs or simply the ~2/3-bigger corpus mean eventually the misfires still change every vector.)
In fact, while I'd early thought all my tests with sample=0 passed, in fact there are anomalies there, too. For example, consider this test where the dummy docs each have a single, unique word, which should always be available for training with no downsampling:
>>> test_d2v_docvecs_trained(doc_args=dict(shared_word=False, count=1001), d2v_args=dict(min_count=0, sample=0, vector_size=4, epochs=1, workers=1, dm=0, seed=42))
.... (25,
[array([0]),
array([1]),
array([2]),
array([3]),
array([4]),
array([6]),
array([7]),
array([8]),
array([10]),
array([12]),
array([13]),
array([14]),
array([15]),
array([18]),
array([19]),
array([20]),
array([21]),
array([22]),
array([24]),
array([27]),
array([28]),
array([39]),
array([41]),
array([53]),
array([61])])This result is especially odd in that all the unadjusted vectors are so early in the 1001.
Another anomaly, when training with a degenerate corpus of documents that are all just the same single word (with no downsampling):
>>> test_d2v_docvecs_trained(doc_args=dict(doc_word=False, count=1001), d2v_args=dict(min_count=0, sample=0, vector_size=4, epochs=1, workers=1, dm=0, seed=42))
... (1, [array([0])])In such an odd corpus, there are never any true 'negative' examples, because any random word chosen will be the same as the target-word. But the positive-training should still happen, and seems to happen for the last 1000 documents - but not document #0.
So lots of oddities here needing deeper investigation.
Investigating the user report at https://groups.google.com/d/msg/gensim/XbH5Sr6RBcI/w5-AIwpSAwAJ, I ran with a (much-smaller) version of the synthetic-corpus there, & reproduced similarly inexplicable results, with doc-vectors that should have received at least some training-adjustment showing no change after an epoch of training.
To demonstrate:
The return value of this test method should, for the given parameters, be a count of 0 unchanged vectors, and and empty-list of unchanged vector indexes. But in a typical run I'm getting instead:
Though this is an artificial dataset, with peculiar 2-word documents, which will often only be 1-word documents (after frequent-word-downsampling of the term shared by all documents) – every document should be at least 1 word long, and thus get at least some training in any single epoch. The logging output regarding sampling accurately reports what the effects should be (at this sampling level):
I get similar evidence of unchanged vectors in
dm=0(PV-DBOW) mode. However, running more epochs usually drives the number of unchanged vectors to 0.Turning off downsampling with
sample=0ensures all vectors show some update, implying some error in the downsampling is involved. Essentially, that strongly implies something going wrong in the code at or around:https://github.com/RaRe-Technologies/gensim/blob/3d6596112f8f1fc0e839a32c5a00ef3d7365c264/gensim/models/doc2vec_inner.pyx#L344
But, a manual check of the precalculated
sample_intvalues for all-but-the-most-frequent word suggests they're where they should be: a value that the random-int is never higher-than, and thus a value that should result in the corresponding words never being down-sampled.I may not have time to dig deeper anytime soon, so placing this recipe-to-reproduce & key observations so far here.
Notably,
Word2Vec&FastTextmay be using similar sampling logic – so even though there's not yet a sighting there, similar issues may exist.(Separately, I somewhat doubt this sample-related anomaly, whatever its cause, is necessarily related to actual original problem of the user in the thread referenced – which seemed only present in extremely large corpuses, likely with real many-word texts, over a normal number of repeated epochs, and perhaps only in the "very tail-end" doc-vectors.)