Closed devkral closed 6 months ago
I found the problem:
dolly uses uint8 datatypes and they are not mapped.
I think it is a byte Tensor which maps to uint8 which causes the problems
I was able to follow how I32 is handled and repeat those same changes for I8 to map to ByteStorage, etc. However the next stumbling block is that it doesn't want to load the tokenizer model. I tried renaming tokenizer.json to tokenizer.model but that just results in a parsing error.
Diff of my changes:
brobert@mac llama.cpp % diff convert.py convert_orig.py
45d44
< DT_I8 = UnquantizedDataType('I8')
74d72
< DT_I8: np.dtype(np.int8)
707d704
< ('torch', 'ByteStorage'): LazyStorageKind(DT_I8)
733d729
< 'I8': DT_I8
yeah that is an other issue: the tokens for dolly are in json format and convert handles some other format.
@devkral Did you end up figuring this out?
I could definitely use this as well if anyone's figured it out
yes, see closed PR: https://github.com/ggerganov/llama.cpp/pull/1308
this is the wrong repository :) see here https://github.com/ggerganov/ggml/tree/master/examples/dolly-v2
This issue was closed because it has been inactive for 14 days since being marked as stale.
Prerequisites
I'm trying to use the truly opensource dolly 2.0 model with lama.cpp. For converting the pytorch bin to ggml I use the convert.py script in the repository.
Please answer the following questions for yourself before submitting an issue.
Expected Behavior
./convert.py either fails with a meaningful error or converts the model to ggml
Current Behavior
The process fails with the cryptic message:
KeyError: ('torch', 'ByteStorage')
Environment and Context
Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.
$ lscpu
$ uname -a
Steps to Reproduce
Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.
python ./convert.py --dump ./models/dolly-v2-12b_pytorch.bin
Failure Logs