search

second-state / WasmEdge-WASINN-examples

Apache License 2.0
220 stars 36 forks source link

[Example] ggml: remove bos in llama3 prompt #142

Closed dm4 closed 3 weeks ago

dm4 commented 3 weeks ago

Since we will add BOS token at WasmEdge (here), remove the <|begin_of_text|> in the examples to avoid this warning:

[warning] [WASI-NN] llama.cpp: llama_tokenize_internal: Added a BOS token to the prompt as specified by the model but the prompt also starts with a BOS token. So now the final prompt starts with 2 BOS tokens. Are you sure this is what you want?

juntao commented 3 weeks ago

Hello, I am a code review bot on flows.network. Here are my reviews of code commits in this PR.

Commit 6a569c6181ed4d27f4b8e83d8d2de479d296c635

Key Changes

  1. In llama-stream/src/main.rs file, the <|begin_of_text|> part was removed from the saved_prompt assignment.

  2. The size of the wasmedge-ggml-llama-stream.wasm file was reduced from 1830057 to 1828839 bytes.

  3. In llama/src/main.rs file, the <|begin_of_text|> part was removed from the saved_prompt assignment.

  4. The size of the wasmedge-ggml-llama.wasm file was reduced from 1826840 to 1825593 bytes.

Potential Problems

  1. Binary Patch: The patch contains binary data, which may cause issues during application of the patch or version control conflicts.

  2. Loss of Information: The removal of <|begin_of_text|> from the prompt assignment could potentially impact the functionality or output expected from the application. This change should be carefully reviewed to ensure it aligns with the intended behavior.

  3. File Size Changes: While reducing file sizes can be beneficial for performance, it's important to verify that the reduction hasn't affected any crucial functionality or data within the wasm files.

  4. Maintainability: The changes made should have corresponding documentation or comments explaining the reason behind the modifications to aid future developers or maintainers.

  5. Impact Analysis: It would be good to assess the impact of these changes on other parts of the codebase or system to ensure overall stability and correctness after the modifications.

  6. Version Control: These changes should be logically connected to an issue or feature to provide context and traceability in the version control system.

It's recommended to carefully test these changes in a staging environment to ensure they work as expected and do not introduce any regressions in the application.