Open TopIdiot opened 1 year ago
@byshiue Could you share some solutions for debugging?
@byshiue it seems like when start_ids is long, the key/value cache for GptNeoxDecoder are wrong. The key/value cache are calculated in GptNeoxContextDecoder, but it is hard to debug. Do you have any suggestions?
@byshiue it seems like when start_ids is long, the key/value cache for GptNeoxDecoder are wrong. The key/value cache are calculated in GptNeoxContextDecoder, but it is hard to debug. Do you have any suggestions?
Hi, @TopIdiot. Thank you for the feedback. You can try printing some intermediate results and compare with HF.
@byshiue Thanks for your reply. Are there something that limit the input length? In this case, when input_ids_length <= 1472, the result is true, and all the key/ value cache for the first input_ids_length are same. But when it come to 1473, the result is wrong. All the key/ value cache is different far away from the key/ value cache when input_ids_length <= 1472.
@byshiue Thanks for your reply. Are there something that limit the input length? In this case, when input_ids_length <= 1472, the result is true, and all the key/ value cache for the first input_ids_length are same. But when it come to 1473, the result is wrong. All the key/ value cache is different far away from the key/ value cache when input_ids_length <= 1472.
We have no idea now. Need to investigate the reason when we have bandwidth.
Branch/Tag/Commit
main
Docker Image Version
nvcr.io/nvidia-pytorch:22.07-py3
GPU name
A100
CUDA Driver
450.156.00
Reproduced Steps
The bug can be reproduced in both fp16 and fp32 mode. As we known, GPTNeoX support max length with 2048. In this example, the length of input ids is 1224, and the length of new tokens is 50, which much smaller than 2048