Integrate Linear and DynamicNTK Scaling Rotary position encoding - Githubissues

OpenBioML / protein-lm-scaling

Other

54 stars 15 forks source link

Integrate Linear and DynamicNTK Scaling Rotary position encoding #44

Closed csjackson0 closed 9 months ago

csjackson0 commented 9 months ago

This PR is a follow up to #34 and integrates Linear and DynamicNTK Scaling Rotary position encoding.

Created a new file under modeling/utils called "scaled_rope_embedding.py"
Updated APT model class to support rotary variants
Added unit tests in the test_encoding.py file for rotary variants
Added rope_scaling_factor and rope_theta to the config

Created test tensors using torch.ones(1,12,10,64) tensor as input into the original implementations at https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L128