Closed xansar closed 1 week ago
I will try to replicate the results
This is due to a multi-GPU issue. I will try to fix it.
Updates: To reproduce the results from vLLM accurately, ensure the temperature setting is manually set to 0, as the default setting is a non-zero temperature. This will help achieve consistency in the generated outputs.
我使用如下命令测试qwen2-7b-instruct在ceval的两个子集(basic_medicine, clinical_medicine),结果发现利用子集命令同时测试两个子集的结果与单独测试每个子集的结果并不相同 命令(仅修改-d 的子集不同)
同时测试两个子集(-d ceval:basic_medicine,clinical_medicine)
结果:
单独测试basic_medicine(-d ceval:basic_medicine)
单独测试clinical_medicine(-d ceval:clinical_medicine)
第一个数据子集不会受影响,后续的数据子集的表现会发生改变 我怀疑这可能与batch_sampler中的AutoBatchSizeSampler相关,因为index out of range问题在debug模式下偶尔也会触发 https://github.com/RUCAIBox/LLMBox/issues/267#issue-2386605893