search

mybigday / llama.rn

React Native binding of llama.cpp
MIT License
257 stars 21 forks source link

llama.cpp running on iPhone generate irrelevant outputs #78

Closed hubertwang closed 2 weeks ago

hubertwang commented 2 weeks ago

Hi everyone,

I tried the example app, and runs it on my devices. On android devices, it works perfectly without any issue. But on iPhone, it generate irrelevant outputs.

(Model: gemma-2-2b-it-Q8-0.gguf) For example, when I ask: "What are the three laws of robotics?" It answered:

You're talking about **data visualization**, also known as "the art of living" - it sounds like something out of an action movie! 💥

Let me know what you need help with! 😄

I am so excited to finally be able to share my work on this platform.  It’s important for everyone in our community who may not have access to resources or opportunities that would otherwise be unavailable. This could include providing financial assistance, legal aid, mental health services, and substance abuse treatment centers near me.”

Here's how I think about AI safety.**

**Introduction:**

The world is facing unprecedented challenges. From climate change to pandemics, we can see that there is a significant difference in performance between two models: A linear model and a decision tree were used to analyze data from a large-scale experiment was conducted at [location] ](https://www.youtube.com/watch?v=h4r30k.png" alt="Image description here." 

Please note: The information provided below is intended for informational purposes only and does not constitute medical advice. Please consult your physician before making any changes to your diet plan. It should contain all relevant details about the product being sold. 

For example, if you are looking for a new home, right? Well, let me tell you a story...

In the heart of bustling London, lies an inconspicuous building tucked away in a quiet corner of town.

This isn't quite what I had in mind! 😄
<end_of_turn><eos>

Looks like the input and output are not related. Anyone has the same issue?

hubertwang commented 2 weeks ago

Ah, sorry, I just noticed that, in the example of README, there's a ==n_ctx== parameter But it is absent in the example code. When I added it to initLlama, it start generating meaningful response.

Not sure why android don't need this, but anyway I found the solution.

    initLlama({
      model: file.uri,
      use_mlock: true,
      n_ctx: 2048,              // added this for iOS
      n_gpu_layers: Platform.OS === 'ios' ? 0 : 0, // > 0: enable GPU
      // embedding: true,
    })