Open kyteinsky opened 1 month ago
fairydreaming
commented
1 month ago @kyteinsky Inference with T5-like models requires using some new API functions (e.g. llama_encode()). Without this not only the llama_cpp_python, but all other software based on llama.cpp won't be able to run encoder-decoder models like T5. I think you should report this bug to the python module.
kyteinsky
commented
1 month ago hey @fairydreaming, thanks for the T5 implementation! Which python module do you refer to? I was under the impression that llama-cpp-python is the python wrapper to use. Also, it doesn't depend on any such package related to llama.cpp.
fairydreaming
commented
1 month ago @kyteinsky Sure, is the python wrapper to use, but it's like a completely separate project from llama.cpp. It's not a part of llama.cpp project, so I think feature requests like support for encoder-decoder T5 models should be reported here: https://github.com/abetlen/llama-cpp-python
kyteinsky
commented
1 month ago That's where we are :)
fairydreaming
commented
1 month ago @kyteinsky lol, didn't notice that all that time, sorry
mtsdurica
commented
1 month ago Same problem here, really hoping the T5 architecture will get implemented soon!
Sadeghi85
commented
1 month ago @fairydreaming
Hi,
Can you help with the python code? I tried these, but output is gibberish:
I added these functions to "_internals.py" (https://github.com/abetlen/llama-cpp-python/blob/main/llama_cpp/_internals.py#L354):
def llama_model_has_encoder(self, model: _LlamaModel):
return llama_cpp.llama_model_has_encoder(
model
)
def encode(self, batch: "_LlamaBatch"):
assert self.ctx is not None
assert batch.batch is not None
return_code = llama_cpp.llama_encode(
self.ctx,
batch.batch,
)
if return_code != 0:
raise RuntimeError(f"llama_encode returned {return_code}")then these lines in "llama.py" (https://github.com/abetlen/llama-cpp-python/blob/main/llama_cpp/llama.py#L633):
if self._ctx.llama_model_has_encoder(self._model.model):
self._ctx.encode(self._batch)
self._ctx.decode(self._batch)@Sadeghi85 One thing that is missing in your code is a preparation of the input to decode() call. Check how it's done in llama-cli source code:
So before calling decode() you have to call llama_model_decoder_start_token() and prepare a batch containing only this single token (assuming that it contains only a single sequence). This will be the input to decode() call.
Sadeghi85
commented
1 month ago I tried, but I really don't know what I'm doing. Waiting for @abetlen to take a look at this.
yugaljain1999
commented
2 weeks ago @Sadeghi85 Have you been able to fixed this issue? cc @fairydreaming @abetlen
@Sadeghi85 One thing that is missing in your code is a preparation of the input to
decode()call. Check how it's done in llama-cli source code:So before calling
decode()you have to callllama_model_decoder_start_token()and prepare a batch containing only this single token (assuming that it contains only a single sequence). This will be the input todecode()call.
fairydreaming
commented
2 weeks ago I see there are serious problems with using T5, so I added a branch with a high-level example of inference with T5 model:
https://github.com/fairydreaming/llama-cpp-python/tree/t5
Also there is a second branch containing low-level example of T5 inference:
https://github.com/fairydreaming/llama-cpp-python/tree/fix-low-level-examples
Hope it helps.
yugaljain1999
commented
2 weeks ago @fairydreaming Thanks a lot for early response and scripts you provided.
May I know can we pass batch prompts as well rather than passing single prompt to model?
If yes, how can we do that? Your response would be highly appreciable.
fairydreaming
commented
2 weeks ago @yugaljain1999 Yes, you can pass multiple prompts. I don't know how it works in llama-cpp-python high-level API, but in llama.cpp (low-level API) you do it by creating a batch containing tokens with different seq_id.
For example if you have two prompts tokenized to pt1 and pt2 token arrays then you create a batch and:
batch.n_tokens = len(pt1) + len(pt2)
batch_index = 0
for i in range(len(pt1)):
batch.token[batch_index] = pt1[i];
batch.pos[batch_index] = i
batch.n_seq_id[batch_index] = 1
batch.seq_id[batch_index][0] = 0 # first prompt has sequence id 0
batch.logits[batch_index] = False
batch_index += 1
for i in range(len(pt2)):
batch.token[batch_index] = pt2[i];
batch.pos[batch_index] = i
batch.n_seq_id[batch_index] = 1
batch.seq_id[batch_index][0] = 1 # second prompt has sequence id 1
batch.logits[batch_index] = False
batch_index += 1then you call llama_encode() on this batch.
For llama_decode you initialize the batch with decoder_start_token for both sequences:
batch.n_tokens = 2
for i in range(2):
batch.token[i] = llama_cpp.llama_model_decoder_start_token(model);
batch.pos[i] = 0
batch.n_seq_id[0] = 1
batch.seq_id[i][0] = i # first token has seq_id 0, second has seq_id 1
batch.logits[i] = Truethen you call llama_decode() on this batch, sample next tokens for both sequences, add them to the batch, call llama_decode() again to generate next tokens for both sequences etc.
yugaljain1999
commented
2 weeks ago @fairydreaming Thanks for this info. Though I was thinking if we could use llama-server for scalable parallel batch inference. But when I try this command, got below error,
Command -
./llama-server -m ../madlad400-3b-mt.gguf -np 16 --temp 0 -ngl -1
Error -
examples/server/server.cpp:693: GGML_ASSERT(llama_add_eos_token(model) != 1) failed
./llama-server(+0x25c87b)[0x6448e414e87b]
./llama-server(+0x25e717)[0x6448e4150717]
./llama-server(+0x4775ff)[0x6448e43695ff]
./llama-server(+0x63fe8)[0x6448e3f55fe8]Any idea how can we resolve it? Any help would be appreciable.
Thanks
fairydreaming
commented
2 weeks ago @yugaljain1999 T5 models are still not supported in llama-server
yugaljain1999
commented
1 week ago @fairydreaming thanks for informing.
As llama-server won't work with T5 so now I am continuing working with low level api example of T5 as you shared, may I know what params we need to change in this script to leverage gpu acceleration fully?
Thanks
I see there are serious problems with using T5, so I added a branch with a high-level example of inference with T5 model:
https://github.com/fairydreaming/llama-cpp-python/tree/t5
Also there is a second branch containing low-level example of T5 inference:
https://github.com/fairydreaming/llama-cpp-python/tree/fix-low-level-examples
Hope it helps.
Prerequisites
Please answer the following questions for yourself before submitting an issue.
Expected Behavior
I'm trying to use models based on the T5 architecture, like "google/flan-t5-small" and "google/madlad400-3b-mt" after converting them to gguf format (8 bit). It works nicely with
llama-cli.It was a recent addition to llama.cpp: https://github.com/ggerganov/llama.cpp/pull/8141
Related issue but I don't understand what needs to be changed here in the python package: https://github.com/ggerganov/llama.cpp/issues/8398
Current Behavior
It loads the model into gpu/memory but the execution fails with a GGML_ASSERT and
(core dumped)Environment and Context
I'm running v0.2.82 using the compiled whl provided for cuda 12.4.
$ lscpu$ uname -aFailure Information (for bugs)
Please help provide information about the failure if this is a bug. If it is not a bug, please remove the rest of this template.
Steps to Reproduce
Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.
python convert_hf_to_gguf.py ./flan-t5-small --outtype q8_0)Note: Many issues seem to be regarding functional or performance issues / differences with
llama.cpp. In these cases we need to confirm that you're comparing against the version ofllama.cppthat was built with your python package, and which parameters you're passing to the context.Failure Logs
It is always the same.