pass@1 score extremely low using GPT-fast API

pytorch-labs / gpt-fast

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

BSD 3-Clause "New" or "Revised" License

5.35k stars 484 forks source link

pass@1 score extremely low using GPT-fast API #94

Closed yafehlis closed 5 months ago

yafehlis commented 5 months ago

Hi:

When I run Phind CodeLlama 34B model (https://huggingface.co/Phind/Phind-CodeLlama-34B-v2) on HumanEval through GPT-fast API, the results give a pass@1 score around 23%. However, the pass@1 score should be above 72%. Could you help me out?

Thanks and Regards, Yao Fehlis (yao.fehlis@amd.com)

yafehlis commented 5 months ago

@kit1980 @msaroufim @yifuwang @huntzhan Could you please help me? Thanks in advance.

yafehlis commented 4 months ago

I have figured the issue out. It is due to PyTorch and ROCm versions.