avg_first_lastshould be the mean of the first and last layers,but hidden_states[0] is the input embedding, hidden_states[1]is the first layer output, should't it be first_hidden = hidden_states[1]??
Thanks for pointing this out! You are right that 0-index is the embedding layer and it has been fixed now. Though it does not affect our results in the paper since they all use CLS or last layer average.
https://github.com/princeton-nlp/SimCSE/blob/main/simcse/models.py#L72-L76
avg_first_last
should be the mean of the first and last layers,buthidden_states[0]
is the input embedding,hidden_states[1]
is the first layer output, should't it befirst_hidden = hidden_states[1]
??