search

bigscience-workshop / petals

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
https://petals.dev
MIT License
8.89k stars 489 forks source link

whether past_key_values can be obtained #526

Open sa1utyeggs opened 8 months ago

sa1utyeggs commented 8 months ago

Hi there. I'm trying to combine chat.petals.dev with streaming-llm (https://github.com/mit-han-lab/streaming-llm).

However, when streaming-llm needs the output.past_key_values, I found that petals returns [RemotePastKeyValues]. image

And the annotation says 'A mock class representing the fact that past_key_values do exist but are stored on remote servers.'.

SO, I wonder if there is any way to get the real value of 'past_key_values'. like this img : image

looking for your reply :)

borzunov commented 8 months ago

Hi @sa1utyeggs,

The actual past_key_values are stored on servers and can't be viewed from clients at the moment. In theory, it's possible to extend the server to send them when necessary (e.g., when a user reads fields of the RemotePastKeyValues object). Alternatively, it may be possible to extend the transformer block code to perform operations you're interested in on servers directly, without sending extra data to the clients.