search

ggerganov / llama.cpp

LLM inference in C/C++
MIT License
63.84k stars 9.15k forks source link

[feature request] Ability to import/export sessions from the UI. #8220

Closed 0wwafa closed 5 days ago

0wwafa commented 1 month ago

It would be useful to be able to import/export the full session from the UI. In this way a session started with a model could be continued with another or reimported with the same model to continue it some other time. I noticed in the javascript console there is an object containing everything. It could be enough to save and restore that. What do you think?

dspasyuk commented 1 month ago

@0wwafa There are UIs based on llama.cpp that are doing that already, ollama, llama.cui just to name a few https://github.com/dspasyuk/llama.cui Look at the list of UIs that are based/have support for the llama.cpp in Readme.

0wwafa commented 1 month ago

thanks.. I just said it would be nice to have it in llama.cpp.

dspasyuk commented 1 month ago

@0wwafa If you just want to save prompts and generation you can set --prompt-cache file.txt and --prompt-cache-all in llama-server it will save it to a file.

ngxson commented 1 month ago

I'm not familiar with the new server UI, but probably POST /slots/:id_slot is not yet implemented on the UI.

Please note that this endpoint is to save/restore state file saved on server side. We don't provide solution for downloading/uploading state via API due to security and stability concern (uploading big file is quite challenge)

0wwafa commented 1 month ago

I'm not familiar with the new server UI, but probably POST /slots/:id_slot is not yet implemented on the UI.

Please note that this endpoint is to save/restore state file saved on server side. We don't provide solution for downloading/uploading state via API due to security and stability concern (uploading big file is quite challenge)

Nope. I was saying to save the "state" on the client, by downloading/uploading a file.

0wwafa commented 1 month ago

@0wwafa If you just want to save prompts and generation you can set --prompt-cache file.txt and --prompt-cache-all in llama-server it will save it to a file.

hmm.. no.. I wish the session object to be saved on the client by clicking a button and uploaded at the start is you want to resume that session.

dspasyuk commented 1 month ago

@0wwafa Resuming the state is possible using prompt embedding but it gets VRAM expensive very quickly. I have not introduced it in llama.cui but it is doable. Llama.cui just loads the previous prompts and responses but does not feed back to the model.

0wwafa commented 1 month ago

@0wwafa Resuming the state is possible using prompt embedding but it gets VRAM expensive very quickly. I have not introduced it in llama.cui but it is doable. Llama.cui just loads the previous prompts and responses but does not feed back to the model.

hmm there must be a "quick" way to do it like loading/saving the session and the token cache/status like from the llama binary but from the webapp.

github-actions[bot] commented 5 days ago

This issue was closed because it has been inactive for 14 days since being marked as stale.