As described in problem #1 and discussion #80, it is not possible to quantize a converted f32.bin under Windows when falcon_quantize is run from the directory where the f32.bin file is located.
In this case, the filename is passed as a relative path that contains nothing but the filename. Therefore, cutting the path by searching for the last \ or / fails, leaving the filename as the "path" to which the /tokenizer.json is appended. So the program does not search in the same directory as the filename, but in a subdirectory with the same name as the file. A simple hack would be to put the tokenizer.json in an appropriately named subdirectory, but on Windows it is not possible to name a directory with the same name as a file in that directory.
This PR fixes that problem and allows the program to find the tokenizer.json in the same directory as the file.
As described in problem #1 and discussion #80, it is not possible to quantize a converted f32.bin under Windows when falcon_quantize is run from the directory where the f32.bin file is located. In this case, the filename is passed as a relative path that contains nothing but the filename. Therefore, cutting the path by searching for the last
\or/fails, leaving the filename as the "path" to which the /tokenizer.json is appended. So the program does not search in the same directory as the filename, but in a subdirectory with the same name as the file. A simple hack would be to put the tokenizer.json in an appropriately named subdirectory, but on Windows it is not possible to name a directory with the same name as a file in that directory.This PR fixes that problem and allows the program to find the tokenizer.json in the same directory as the file.