Interruptible Async Model Loading With Progress Monitoring

SciSharp / LLamaSharp

A C#/.NET library to run LLM (🦙LLaMA/LLaVA) on your local device efficiently.

https://scisharp.github.io/LLamaSharp

MIT License

2.18k stars 294 forks source link

Interruptible Async Model Loading With Progress Monitoring #702

Closed martindevans closed 1 month ago

martindevans commented 1 month ago

Added LLamaWeights.LoadFromFileAsync.
Updated examples to use LoadFromFileAsync
Async loading supports cancellation through a CancellationToken:
- If loading is cancelled an OperationCanceledException is thrown.
- If it fails for another reason a LoadWeightsFailedException is thrown.

This does not expose the progress callback on the ModelParams object. However the code as written does support custom callbacks being set so that can be easily exposed in the future.

Resolve #699

martindevans commented 1 month ago

I added an IProgress parameter to the LoadFromFileAsync from method as well.

It's not _quite) reporting the raw values. If there are LoRAs to apply it reports model loading in the [0 -> 0.9] range and then reports LoRA loading progress in the [0.9 -> 1.0] range.