netdur / llama_cpp_dart

dart binding for llama.cpp
MIT License
169 stars 22 forks source link

Use package:typed_isolate for easier isolate management #43

Closed Levi-Lesches closed 1 day ago

Levi-Lesches commented 1 week ago

Hi @netdur! I saw your comment on Reddit about isolates, and I saw a reply offering to rewrite that part with package:actors, so as author of package:typed_isolate I thought I'd step in and give it a shot.

Basically, instead of managing the main isolate and model isolate in one class, I created three files:

I wasn't sure if you were using to/from JSON for anything else, but if not, you can remove those too. Feel free to ignore this, but hopefully it makes your code clearer! I'm also looking for feedback for the package so if you see anything you wish were different please let me know!

netdur commented 1 week ago

Thank you @Levi-Lesches I am in middle of updating package with llama.cpp I will review your improvements

netdur commented 2 days ago

@Levi-Lesches I've pushed my work-in-progress changes to the dev branch, primarily focusing on sampler implementations. Your PR should be compatible with minimal adjustments needed. Could you please review and confirm everything aligns properly? Thank you!

Levi-Lesches commented 1 day ago

@netdur I merged and updated to the best of my understanding, but granted, I'm just filling in missing context from nearby code. I left three TODO comments explaining where some fields were removed and that previous code is no longer compatible but I assume that's intentional and you have a replacement.

Overall, you'll need to review and test to make sure I got the Llama-specific parts right, but yes, this parent/child relationship should work for your project.

netdur commented 22 hours ago

@Levi-Lesches I have merged your code, thank you I have implemented missing functions, you can now resold todo list can you make an example use?

Levi-Lesches commented 13 hours ago

Sorry, I'm not really knowledgeable on LLMs and Llama or how exactly your app is meant to be tested. I just wanted to help make your isolate code simpler for future use.

If you have any more isolate-related questions, feel free to reach out again though!