Crash (native stack) for llama2 demo on Android

[AmarOk1412]

Environment

Followed: https://pytorch.org/executorch/stable/llm/llama-demo-android.html

Downloaded llama2-chat (7B) and run the android demo (Pixel 6, Android 14).

Using commit: 088cedfb9beb45ee4a66759e7bddc52e9366989b (HEAD, tag: v0.2.1-rc5, tag: v0.2.1, origin/release/0.2)

Reproduce steps

1. Run the demo
2. Input:

   <s>
   [INST]
   <<SYS>>
   You're a parser. Your role is to take a sentence in argument, parse it and output a minized JSON and only a JSON (NO OTHER SENTENCE).
   The sentence can contains several ingredients (with optional quantity and units)
   E.g: "50ml of milk" = {"ingredients":[{"ingredient": "milk", "quantity": 50, "unit": "mL"}]}
   Or: "salt and pepper" = {"ingredients":[{"ingredient": "salt"},{"ingredient":"pepper"}]}
   Avoid useless words e.g.: "salt as you wish" = {"ingredients":[{"ingredient": "salt"}]}.
   Units should be metrics or for american kitchen (like cups, oz, cL, mL)
   Sentences can be from French or english
   <</SYS>>
   "salt and pepper"
   [/INST]

3. Generated => Crash

Output

Crash:

2024-07-05 10:53:47.789 31097-31179 libc                    com.example.executorchllamademo      A  Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 31179 (Thread-3), pid 31097 (utorchllamademo)
---------------------------- PROCESS STARTED (31186) for package com.example.executorchllamademo ----------------------------
2024-07-05 10:53:48.867 31183-31183 DEBUG                   crash_dump64                         A  Cmdline: com.example.executorchllamademo
2024-07-05 10:53:48.867 31183-31183 DEBUG                   crash_dump64                         A  pid: 31097, tid: 31179, name: Thread-3  >>> com.example.executorchllamademo <<<
2024-07-05 10:53:48.867 31183-31183 DEBUG                   crash_dump64                         A        #01 pc 00000000035c4624  /data/app/~~m7Cz0fiHuS0Pomw97DvFSQ==/com.example.executorchllamademo-Bo37H0Ldn-EE8QF5dXCgcg==/lib/arm64/libexecutorch_llama_jni.so (et_pal_abort+8) (BuildId: 39b8648fd1a373ba9a341709299c076d41970345)
2024-07-05 10:53:48.867 31183-31183 DEBUG                   crash_dump64                         A        #02 pc 00000000035c45d4  /data/app/~~m7Cz0fiHuS0Pomw97DvFSQ==/com.example.executorchllamademo-Bo37H0Ldn-EE8QF5dXCgcg==/lib/arm64/libexecutorch_llama_jni.so (torch::executor::runtime_abort()+8) (BuildId: 39b8648fd1a373ba9a341709299c076d41970345)
2024-07-05 10:53:48.867 31183-31183 DEBUG                   crash_dump64                         A        #03 pc 00000000035b7f78  /data/app/~~m7Cz0fiHuS0Pomw97DvFSQ==/com.example.executorchllamademo-Bo37H0Ldn-EE8QF5dXCgcg==/lib/arm64/libexecutorch_llama_jni.so (torch::executor::Runner::generate(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&, int, std::__ndk1::function<void (std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&)>, std::__ndk1::function<void (torch::executor::Runner::Stats const&)>)+4384) (BuildId: 39b8648fd1a373ba9a341709299c076d41970345)
2024-07-05 10:53:48.867 31183-31183 DEBUG                   crash_dump64                         A        #04 pc 000000000012fefc  /data/app/~~m7Cz0fiHuS0Pomw97DvFSQ==/com.example.executorchllamademo-Bo37H0Ldn-EE8QF5dXCgcg==/lib/arm64/libexecutorch_llama_jni.so (executorch_jni::ExecuTorchLlamaJni::generate(facebook::jni::alias_ref<_jstring*>, facebook::jni::alias_ref<executorch_jni::ExecuTorchLlamaCallbackJni>)+124) (BuildId: 39b8648fd1a373ba9a341709299c076d41970345)
2024-07-05 10:53:48.867 31183-31183 DEBUG                   crash_dump64                         A        #05 pc 000000000013019c  /data/app/~~m7Cz0fiHuS0Pomw97DvFSQ==/com.example.executorchllamademo-Bo37H0Ldn-EE8QF5dXCgcg==/lib/arm64/libexecutorch_llama_jni.so (facebook::jni::detail::MethodWrapper<int (executorch_jni::ExecuTorchLlamaJni::*)(facebook::jni::alias_ref<_jstring*>, facebook::jni::alias_ref<executorch_jni::ExecuTorchLlamaCallbackJni>), &(executorch_jni::ExecuTorchLlamaJni::generate(facebook::jni::alias_ref<_jstring*>, facebook::jni::alias_ref<executorch_jni::ExecuTorchLlamaCallbackJni>)), executorch_jni::ExecuTorchLlamaJni, int, facebook::jni::alias_ref<_jstring*>, facebook::jni::alias_ref<executorch_jni::ExecuTorchLlamaCallbackJni> >::dispatch(facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch_jni::ExecuTorchLlamaJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::alias_ref<_jstring*>&&, facebook::jni::alias_ref<executorch_jni::ExecuTorchLlamaCallbackJni>&&)+88) (BuildId: 39b8648fd1a373ba9a341709299c076d41970345)
2024-07-05 10:53:48.867 31183-31183 DEBUG                   crash_dump64                         A        #06 pc 00000000001300a4  /data/app/~~m7Cz0fiHuS0Pomw97DvFSQ==/com.example.executorchllamademo-Bo37H0Ldn-EE8QF5dXCgcg==/lib/arm64/libexecutorch_llama_jni.so (facebook::jni::detail::FunctionWrapper<int (*)(facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch_jni::ExecuTorchLlamaJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::alias_ref<_jstring*>&&, facebook::jni::alias_ref<executorch_jni::ExecuTorchLlamaCallbackJni>&&), facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch_jni::ExecuTorchLlamaJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*, int, facebook::jni::alias_ref<_jstring*>, facebook::jni::alias_ref<executorch_jni::ExecuTorchLlamaCallbackJni> >::call(_JNIEnv*, _jobject*, _jstring*, facebook::jni::detail::JTypeFor<executorch_jni::ExecuTorchLlamaCallbackJni, facebook::jni::JObject, void>::_javaobject*, int (*)(facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch_jni::ExecuTorchLlamaJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::alias_ref<_jstring*>&&, facebook::jni::alias_ref<executorch_jni::ExecuTorchLlamaCallbackJni>&&))+84) (BuildId: 39b8648fd1a373ba9a341709299c076d41970345)
2024-07-05 10:53:48.868 31183-31183 DEBUG                   crash_dump64                         A        #12 pc 0000000000ebfb18  /data/app/~~m7Cz0fiHuS0Pomw97DvFSQ==/com.example.executorchllamademo-Bo37H0Ldn-EE8QF5dXCgcg==/oat/arm64/base.vdex (com.example.executorchllamademo.MainActivity$2.run+0)
---------------------------- PROCESS ENDED (31097) for package com.example.executorchllamademo ----------------------------

However in native debug I didn't get good results to dig more

[kirklandsign]

Hi @AmarOk1412 do you have the logs for the system? Any logs from lmkd? With Pixel 6 it might get OOM killed

[AmarOk1412]

2024-07-06 14:58:59.log Sure here is the log

I don't show any log for lmkd

[kirklandsign]

@AmarOk1412 Thank you for the updates! So you see no output tokens at all, and it just crashed?

[AmarOk1412]

Yes exactly.