Closed nakkapeddi closed 6 months ago
Hmm, can you try --quant=false
without the double quotes? I think the double quote might be problematic.
Looks like simply setting --quant=false
didn't work.
I pushed a fix here. The solution is to make --quant
an bool flag by setting the action='store_true'
. Please let us know if that work for you. Thanks!
https://github.com/google/gemma_pytorch/commit/5f2c166bd7bdd24daa868a827cfbec3bb5405ef3
Replace parser.add_argument("--quant", type=bool, default=False)
in run.py
with parser.add_argument("--quant", action='store_true')
. As a result, if you include --quant
on the command line, args.quant
will be set to True
. If you do not include --quant
, then args.quant
will be set to False
. Here is an example of using the quantified gemma model:
python scripts/run.py --device=cuda --ckpt=/tmp/ckpt --variant="2b" --quant --prompt="Hi, gemma."
Thanks for the fix, Pengchong! Looks like this can be closed.
Regardless of what value is passed in, the --quant argument will always return True. Python will cast the string to always true, even if you have set it to false:
Solution is to have a portable strtobool utility, and I suggest having a contract of something like "0", "1", "true", "false". Bash is stringly typed and Python is duck typed, so it leads to this kind of issues.