search

shamblett / alpaca

An alpaca implementation in Dart
MIT License
5 stars 0 forks source link

Optimise the Dart code #3

Open shamblett opened 1 year ago

shamblett commented 1 year ago

The Dart code should be analysed by dev tools to optimize it's performance.

shamblett commented 1 year ago

There is still some work to do on the memory allocation from the fixes applied on issue 2, this should also be done on this issue.

shamblett commented 1 year ago

Memory allocation issue now fixed, memory use now stable at 11.5GB across multiple queries.

abcnow commented 10 months ago

How fast is dart with this? Since it is one-threaded type of language ?

shamblett commented 10 months ago

Its slower than its C++ counterpart in the VM but on par if you compile the code to an executable. This is because there isn't really a lot of Dart code, all the heavy lifting is done in the ggml library which is accessed via FFI, this library starts and manages its own threads as needed(configurable) so although Dart is single threaded this app isn't as such as far a Linux is concerned.

I think the Dart code is as good as it gets without restructuring it to be me more Dart like and less C++ like although I haven't yet done an exhaustive performance analysis so this is really just guesswork.