Open zhang-ge-hao opened 1 year ago
I seem to have fixed this bug.
param.max_sequence_lengths
will be updated in invokeGatherTree(param);
however, if the sequence_lengths_
was updated, the process in the next step will be wrong.
so it's sequence_lengths
that should be passed into the function invokeGatherTree(param);
instead of sequence_lengths_
.
I'll submit a PR.
@byshiue
Branch/Tag/Commit
main
Docker Image Version
nvcr.io/nvidia/pytorch:21.11-py3
GPU name
TITAN
model
https://huggingface.co/TabbyML/NeoX-1.3B
Reproduced Steps
the code which can reproduce this bug: https://github.com/AkiyamaYummy/FasterTransformer/tree/tmp
the difference between my buggy code and the newest main branch's code: https://github.com/NVIDIA/FasterTransformer/commit/2c046e3a47836073c867b8fbc284ce7b4d68cac4
Got an invalid result when the callback was set.
without
registerCallback()
used, after the gptneox_example run, theout
file was:with
registerCallback()
used, after the gptneox_example run, theout
file was: