search

OpenLLMAI / OpenRLHF

An Easy-to-use, Scalable and High-performance RLHF Framework (70B+ PPO Full Tuning & Iterative DPO & LoRA & Mixtral)
https://openrlhf.readthedocs.io/
Apache License 2.0
1.73k stars 164 forks source link

Compatibility between vllm and NGC #228

Closed THINK2TRY closed 4 months ago

THINK2TRY commented 4 months ago

Hi, thanks for your wonderful work! Is there any version of NGC recommended to run this framework? I tried to build vllm in NGC 23.12 / 24.01 but it reports that vllm is not compatible, as vllm requires torch-2.1.2. So is there any recommendation or solution to this problem?

Many thanks!

hijkzzz commented 4 months ago

I suggest you build a new container image including vLLM and torch-2.1.2~ Contributions to the openrlhf are also welcome

hijkzzz commented 4 months ago

related issue: https://github.com/vllm-project/vllm/pull/2804

THINK2TRY commented 4 months ago

@hijkzzz Thanks for you reply. I will try the upgraded torch-2.2.0 in vllm. Btw, have you ever compared the performance between nemo-aligner and openrlhf? It seems that there is no inference optimization in Nemo-aligner but I'm still curious about the efficiency comparison.

hijkzzz commented 4 months ago

@hijkzzz Thanks for you reply. I will try the upgraded torch-2.2.0 in vllm. Btw, have you ever compared the performance between nemo-aligner and openrlhf? It seems that there is no inference optimization in Nemo-aligner but I'm still curious about the efficiency comparison.

inference with nemo megatron is very slow

hijkzzz commented 4 months ago

We added Dockerfile in https://github.com/OpenLLMAI/OpenRLHF/commit/8773cd137a8146e0312bf5543577deaa16cd14b7