Add RoPE Interpolation - Githubissues

LAION-AI / Open-Assistant

OpenAssistant is a chat-based assistant that understands tasks, can interact with third-party systems, and retrieve information dynamically to do so.

https://open-assistant.io

Apache License 2.0

36.94k stars 3.22k forks source link

Add RoPE Interpolation #3564

Closed shahules786 closed 1 year ago

shahules786 commented 1 year ago

Added support for RopE interpolation via the SuperHOT method and its variants proposed in reddit scaled-rope

Supported methods

Linear scaling
NTK aware scaling
Dynamic NTK

Supported Models

LLAMA
Falcon

This can be easily extended and experimented with by configuring two parameters superhot and superhot_config

github-actions[bot] commented 1 year ago

:x: pre-commit failed. Please run pre-commit run --all-files locally and commit the changes. Find more information in the repository's CONTRIBUTING.md