search

instructlab / instructlab

InstructLab Command-Line Interface. Use this to chat with a model and execute the InstructLab workflow to train a model using custom taxonomy data.
https://instructlab.ai
Apache License 2.0
762 stars 287 forks source link

[Epic - Phase 2 Profiles] Introduce Profiles in `ilab` #2108

Open cdoern opened 3 weeks ago

cdoern commented 3 weeks ago

What is the current experience of ilab you've found issue with? Using Instructlab with different hardware capabilities

How do you think the described experience should be? Users should be able to set different configurations upon initialization of instructlab that allow them to properly use their hardware. These profiles should contain things like the following:

  1. always use vllm or llamacpp based on GPU availability and vRAM
  2. default to certain models based off of hardware availability for different tasks
  3. set certain things like gpus for generate, evaluate, etc based on hardware
  4. download all relevant models (via a prompt) for the user that are in the profile so all commands work out of the box (unless the models are on the system.

What is the benefit of the change you are proposing?

In ilab 0.18.0 the idea of a train-profile was introduced. These yaml files allow users to set different options for training based on their GPU configuration. This idea should be more broadly expanded across ilab in the following ways to alleviate user confusion about how to best utilize the newly introduced more intensive pipelines.

Additional context

The system can look like this:

ilab config init
Found /home/cdoern/.config/instructlab/config.yaml, do you still want to continue? [y/N]: y
Found /home/cdoern/.local/share/instructlab/internal/train_configuration/profiles, do you also want to reset existing profiles? [y/N]: y
Welcome to InstructLab CLI. This guide will help you to setup your environment.
Please provide the following values to initiate the environment [press Enter for defaults]:
Path to taxonomy repo [/home/cdoern/.local/share/instructlab/taxonomy]: 
Path to your model [/home/cdoern/.cache/instructlab/models/merlinite-7b-lab-Q4_K_M.gguf]: 
Generating `/home/cdoern/.config/instructlab/config.yaml` and `/home/cdoern/.local/share/instructlab/internal/train_configuration/profiles`...
Please choose a system profile to use:
[0] Choose for me
[1] Single Consumer GPU (8GB-16GB vRAM)
[2] Multi Consumer GPU (24+GB vRAM)
[3] Single Server GPU (24+GB vRAM)
[4] Multi Server GPU (48+GB vRAM)
[5] MacOS M-Series GPU (16GB+ vRAM)
Enter the number of your choice [hit enter for the default System profile] [0]: 1
Please choose which train profile matches your system configuration:
[0] Choose for me
[1] Nvidia RTX 4070
[2] Nvidia RTX 4080
[3] Nvidia RTX 4090

The first choice would allow the user to get some sane defaults for commands like data generate, model evaluate etc. And then the secondary prompt allows us to set proper effective_batch_size and other entries in the _train config.

This is a pretty lofty EP so I intend to use this issue as a tracker for sub-issues

ktam3 commented 3 weeks ago

additional d/s req

We need to implement a notion of CLI profiles for InstructLab, because at a high level we have laptop vs. big GPU boxen use cases, and finer-grained we have specific GPU / vendor configurations and use cases. Each major component has potential for profiles to simplify usage (particularly training, which we already have an early implementation for.)

General goal is to provide a good UX experience with great output OOTB without a lot of hacking required by end users. Anti-goal is to be an ffmpeg-like experience with a bajillion flags to get a relatively basic action working.

Feature Overview (aka. Goal Summary) I want to be able to get started as soon and as smoothly as possible, without spending a lot of time configuring different parts and pieces.

Goals (aka. expected user outcomes)

CLI users should get a complete configuration file with the relevant changes implemented across different sections (training, serving, generate, eval) based on their GPU configuration, without having to make a lot of changes (i.e. enabling individual flags etc). We want them to select their hardware, do ‘ilab config -init’, copy it over and start working on the next piece.

Questions to Answer (Optional):

How do we extend training profiles to config profiles that cover each component - generate, evaluate, train? How do we build config profiles for different hardware configurations? How do we detect hardware?