While using the BERT processed datasets, for example CMU-MOSI provided by MMSA, I found something confusing.
As the data is composed by ['train', 'valid', 'test'], let's take dataset['train'] as an example:
As we all know, token_id 0 in BERT tokenizer refers to [PAD], of which the meaning is the padding token of the current sequence. Thus, it is supposed to map the exact same semantic vector. However, when I print out the last few vectors of the text embedding sequence, it turns out like this:
Due to the length of the vector, I only showed the first few datas, but you can see they are not the same.
I don't know if I don't understand Bert well enough or if the data is being preprocessed differently.
I am looking forward for your early reply.
While using the BERT processed datasets, for example CMU-MOSI provided by MMSA, I found something confusing. As the data is composed by ['train', 'valid', 'test'], let's take dataset['train'] as an example:
As we all know, token_id
0
in BERT tokenizer refers to[PAD]
, of which the meaning is the padding token of the current sequence. Thus, it is supposed to map the exact same semantic vector. However, when I print out the last few vectors of the text embedding sequence, it turns out like this:Due to the length of the vector, I only showed the first few datas, but you can see they are not the same. I don't know if I don't understand Bert well enough or if the data is being preprocessed differently. I am looking forward for your early reply.