search

LAION-AI / Open-Assistant

OpenAssistant is a chat-based assistant that understands tasks, can interact with third-party systems, and retrieve information dynamically to do so.
https://open-assistant.io
Apache License 2.0
37.04k stars 3.23k forks source link

Configure PEFT from config #3571

Closed shahules786 closed 1 year ago

shahules786 commented 1 year ago

What

Added support to configure PEFT from config and save WTE embeddings with adapter files to enable easy loading of OA Lora weights.

Why

Earlier PEFT modules were hardcoded for llama model only. This was an issue when training other models using peft like RWModel, GPTNeoX, etc

How

Introduces extra parameter peft_config to config.yml

github-actions[bot] commented 1 year ago

:x: pre-commit failed. Please run pre-commit run --all-files locally and commit the changes. Find more information in the repository's CONTRIBUTING.md