Open orhansonmeztr opened 1 year ago
I'm not super familiar with generating document level representations from GPT-2 models, but your code looks like it is summing the hidden states for each position and normalizing? That would be 2560. Another way is to just look at the final hidden state, which would also be 2560. But I would expect the document level vector to be 2560 since you are going to use some algorithm to combine the size=2560 vectors into one final vector.
Could you point to paper, algorithm, code of how you want to generate final abstract-level representations ? As I said, it looks like your method is to add all of the final states and normalize. I think typically one would just take the final hidden state of the sequence.
What task do you want to use these abstract level vectors for ?
Hi. First of all, thank you for making such a model available to us. I am trying to get vector embeddings of abstracts of some of the articles in PubMed. But somehow I couldn't get the sentence embeddings. More precisely, I wrote the code below and the dimensions of the vectors I obtained are 2560. But on the huggingface page, it says sequence length is 1024. So I understand that the dimension of an embedding vector should be 1024. Am I wrong? Can you help with getting sentence embeddings? Best wishes. Orhan