search

guidance-ai / guidance

A guidance language for controlling large language models.
MIT License
19.05k stars 1.04k forks source link

Transformers `past_key_values` deprecated #986

Open hudson-ai opened 2 months ago

hudson-ai commented 2 months ago

With transformers version 4.44.0 (latest at time of writing), I get a warning regarding the use of past_key_values in the TransformersEngine.

We detected that you are passing past_key_values as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate Cache class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache) You are not running the flash-attention implementation, expect numerical differences.

The warning is clearly wrong (as 4.44.0 > 4.43), but if this usage is truly going to be removed, we need to fix our caching logic.

usholanb commented 2 months ago

agree, also T5 from

https://github.com/huggingface/transformers/blob/746104ba6f0514159d58dc2fb09c887d0e9d4863/src/transformers/models/t5/modeling_t5.py#L1650

expects past_key_values as Tuple

Harsha-Nori commented 2 months ago

@usholanb, interesting, thanks for sharing! If I'm reading this right, does that mean that we can't continue to support T5 if we move onto the new transformers.Cache API?

usholanb commented 2 months ago

@Harsha-Nori I don't know. I haven't switched to transformers.Cache myself. Just sharing my concern since this message is making me worried for no clear reason

hudson-ai commented 1 month ago

Getting a new error in CI: "HybridCache not subscriptable" -- may very well be due to them finally making some breaking changes with the Cache API.

https://github.com/guidance-ai/guidance/actions/runs/11050523063