Bug: Evals might be broken in pinned HF transformers version `cache=False`

jzhang38 / EasyContext

Memory optimization and training recipes to extrapolate language models' context length to 1 million tokens, with minimal hardware.

Apache License 2.0

616 stars 42 forks source link

Bug: Evals might be broken in pinned HF transformers version `cache=False` #26

Closed michaelfeil closed 4 months ago

michaelfeil commented 5 months ago

Checked out your evals!

I think this should affect the generation quality. https://github.com/huggingface/transformers/pull/30380

jzhang38 commented 5 months ago

Hi, thanks for the info. If I understand this correctly, it only affects the generate() function? We did not use the generate function in our eval. See https://github.com/jzhang38/EasyContext/issues/19

michaelfeil commented 4 months ago

Ah, gotcha. This should be irrelevant then, you are correct! (We only do 1x prefill)