Open Mohrizk opened 1 month ago
Mohrizk
commented
1 month ago It consistently worked when i manually removed the special token causes this issue [9] with the below line
filtered_tokens = [token for token in out_tokens[0] if token != 9]
I still think we need a valid solution for this, if possible
patrickvonplaten
commented
1 month ago Hey @Mohrizk,
Thanks for the issue - can you please make sure to post a fully, reproducible code snippet that I can copy-paste into a Python shell and it will run correctly.
For the above code snippet, I don't know exactly where you took the model weights from. Also Transformer and MistralTokenizer are not imported so the code snippet is not runable.
Can you try to post a complete code snippet please?
Python -VV
Pip Freeze
Reproduction Steps
I am working on function calling methods, however i face challenged in the decode function as the output has not been consistent sometimes it produces the parameters required and other it errors as specialtoken policy error despite of turning it to ignore "1"
Here is my snippet, while use Nemo-Instruct-2407:
My issue due to output creates fluctuation, most of the time produces the below error:
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer from mistral_common.tokens.tokenizers.tekken import SpecialTokenPolicy
tokenizer = MistralTokenizer.v3(is_tekken=True) tokenizer.special_token_policy = SpecialTokenPolicy.IGNORE # or SpecialTokenPolicy.KEEP
Expected Behavior
Expected is to produce the params of the flow id required
Additional Context
No response
Suggested Solutions
No response