Different vectors for a sentence if it's in a batch

[IoDmitri]

Here is my setup:

model = torch.load('encoder/infersent.allnli.pickle')
model.set_glove_path(<my_local_path_to_glove>)
model.build_vocab_k_words(K=100000)

s1 = "I went to the movies"
s2 = "I'm now at the beach"

vectors = model.encode([s1, s2])

Here's the printout from vectors:

array([[ 0.08746275,  0.07032087,  0.        , ...,  0.        ,
         0.        ,  0.02636627],
       [ 0.03963339,  0.09454132,  0.01724778, ..., -0.00346539,
        -0.03814263,  0.01555013]], dtype=float32)

s1_enc = model.encode([s1])

array([[ 0.08746275,  0.07032087, -0.02096005, ..., -0.03267486,
        -0.02566048,  0.02636627]], dtype=float32)

Now these look similar, but aren't exactly the same.

s2_enc = model.encode([s2])

array([[ 0.03963339,  0.09454132,  0.01724778, ..., -0.00346539,
        -0.03814263,  0.01555013]], dtype=float32)

Now these are exactly the same. I'm wondering why there's a disparity between batching up the sentences and encoding them separately.

[aconneau]

Hi, I am not sure why that happens, I guess it comes from numerical approximations somewhere. It may be linked to the CUDNN implementation of the LSTM which makes the precision not perfect. I would suggest to try to reproduce this behaviour with a simple LSTM and a toy input and ask for explanations on the pytorch forum.

(Closing this issue because probably not directly linked to InferSent)