-
https://github.com/linkedin/Liger-Kernel
Should be easy with https://github.com/linkedin/Liger-Kernel/issues/242#issuecomment-2347334135
Also we should patch `kl_div` loss with ligers kl_div if …
lapp0 updated
1 month ago
-
Per the installation instructions on the wiki:
if(!checkPkg("liger")) install_github("MacoskoLab/liger")
 didn't find any similar reports.
### Exp…
-
Consider implementing the Liger Kernels which has shown to yield large memory savings.
- RoPE: 3X speedup with ~3X peak memory reduction.
- SwiGLU: 1.5X peak memory reduction
- Cross Entropy: >4X…
-
### Please check that this issue hasn't been reported before.
- [X] I searched previous [Bug Reports](https://github.com/axolotl-ai-cloud/axolotl/labels/bug) didn't find any similar reports.
### Exp…
-
https://github.com/linkedin/Liger-Kernel
-
### 🚀 The feature, motivation and pitch
We want to support various alignment and distillation loss functions.
Refer this PR on ORPO: #362
## Progress
### Alignment
- [x] ORPO https://gith…