search

OpenLMLab / LOMO

LOMO: LOw-Memory Optimization
MIT License
978 stars 68 forks source link

installable package #74

Closed Borda closed 8 months ago

Borda commented 8 months ago

Hello, great work on LLM optimization and enabling space reduction for fine-tuning. I was just wondering if you plan to make it more user accessible such as by making the project as Python package that could be installed? :rabbit:

KaiLv69 commented 8 months ago

Hello, thanks for your insterests.

lomo-optim can be installed from PyPI using pip now.

pip install lomo-optim

Then, import Lomo or AdaLomo.

from lomo_optim import Lomo
from lomo_optim import AdaLomo

The usage of Lomo and AdaLomo is similar but not the same as PyTorch's optimizers (example). We recommend to use AdaLomo without gradnorm to get better performance and higher throughput.

Borda commented 8 months ago

addressed in #75