search

guidance-ai / guidance

A guidance language for controlling large language models.
MIT License
19.04k stars 1.04k forks source link

Support for microsoft/phi-1_5 model #373

Closed CorentinvdBdO closed 11 months ago

CorentinvdBdO commented 1 year ago

There's a simple hack to run phi-1_5:

in guidance/llms/transformers._py line 344:

Replace: self._prefix_cache = streamer.generated_sequence[0][:self._past_key_values[0][0].shape[-2]] by self._prefix_cache = streamer.generated_sequence[0][:self._past_key_values.key_value_memory_dict[0][0].shape[-2]]

I don't trust myself to make a proper change that'd not destroy everything so I leave this here for anyone interested.

marcotcr commented 11 months ago

This works in the new release:

lm = models.Transformers('microsoft/phi-1_5', trust_remote_code=True)
lm + 'Hello ' + gen(max_tokens=10)

Hello fellow Redditors! Today, I want to talk,

Sorry it took us so long to get to it!