Open taylorgoolsby opened 6 months ago
I noticed mlx-community/phi-2-hf-4bit-mlx has some custom python code which should be ran when using the python version of MLX. Since this custom code does not run on swift, does the Phi class in LLMEval implement a port of that custom code?
I noticed mlx-community/phi-2-hf-4bit-mlx has some custom python code which should be ran when using the python version of MLX. Since this custom code does not run on swift, does the Phi class in LLMEval implement a port of that custom code?
No, the code has some simple prompt augmentation, but it doesn't use the config:
For the general issue of debugging the tokenizer, since we have a working python version it is probably easiest to compare to that. I wrote up some thoughts here, see if they help:
I am running the phi-2 on iOS using the code from LLMEval.
I have ported over an implementation of the CodeGen Tokenizer into swift as a standalone file:
I have now having an issue generating the response from phi-2. Even though the tokenizer seems to be correct, the response from Phi-2 is incoherent.
For reference here is a snippet of the code which generates tokens:
Here are the print logs:
Can anyone point me in the right direction to fixing this?