[Roadmap] LMFlow Roadmap

[wheresmyhair]

This document includes the features in LMFlow's roadmap. We welcome any discuss or contribute to the specific features at related Issues/PRs. 🤗

Main Features

• Data
  • [ ] DPO dataset format #867
  • [ ] Conversation template in DPO
  • [ ] jinja template
  • [ ] Tools in conversation dataset
• Model
  • Backend
    • [ ] Accelerate support
• Pipeline
  • Train/Finetune/Align
    • [ ] DPO (multi-gpu) #867
    • [ ] Iterative DPO #867
    • [ ] PPO
    • [ ] LISA (multi-gpu, qwen2, chatglm)
    • [ ] Batch size and learning rate recommendation (arxiv)
  • Inference
    • [x] vllm inference #860 #863
    • [ ] Reward model scoring #867
    • [ ] Multiple instances inference (vllm, rm, others)
    • [ ] Inference checkpointing and resume from checkpoints

Usability

• [ ] Inference method auto-downgrading (vllm>ds, etc.), and make vllm package optional
• [ ] Merging similar model methods into hf_model_mixin

Bug fixes

• [ ] model.generate() with dsz3 #861
• [ ] merge_lora lora with abs path merging

Issues left over from history

• [ ] use_accelerator -> use_accelerate typo fix (with Accelerate support PR)
• [ ] model_args.use_lora leads to truncation of the sequence, mentioned in #867
• [ ] Make ports, addresses, and all other settings in distributed training tidy and clear (with Accelerate support PR)

Documentation

• [ ] Approx GPU memory requirement w.r.t model size & pipeline
• [ ] Dev handbook, indicating styles, test list, etc.

[wheresmyhair]

Note on multiple instances inference: In vllm inference, the number of attn heads should be devisible by vllm tensor parallel size. If we have a 14 heads LLM, then the options for tp is 1&2 (7 will cause another division issue, but I just forget what that issue is). Say we have 8 gpus, then to utilize these devices, multiple instances vllm inference is necessary (tp=1 -> 8 instances, and tp=2 -> 4 instances) Also, same for rm inference, and any other inference pipelines.