search

zhenyuhe00 / BiPE

Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation, ICML 2024
MIT License
20 stars 1 forks source link

It seems that bipe_alibi is not working yet. #1

Open liveck opened 1 month ago

liveck commented 1 month ago

It seems that bipe_alibi is not working yet.

get_ape_embeddings returns a tuple, which is different from embed_tokens.

All codes since here do not work.

if self.config.rpe_type == "bipe_alibi":
    inputs_embeds = self.get_ape_embeddings(torch.stack([input_ids, token_ids], dim=-1))
else:
    inputs_embeds = self.embed_tokens(input_ids)
zhenyuhe00 commented 1 month ago

Hi, Sorry about the typo in get_ape_embeddings and thanks for pointing this out.

It should be return embed rather than return embed, X[:, :, 0].