Why is CodeGeeX2-6b much slower than ChatGLM2-6b

[Wallong]

As the title says, on the same graphics card (3090), CodeGeeX2-6b is much slower than ChatGLM-6b. According to the official demo, I would like to know if there are any tricks.

[Wallong]

I find the way to inference in demo/run_demo.py is faster.

inputs = tokenizer([prompt], return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_length=inputs['input_ids'].shape[-1] + request.max_tokens
）

instead of

inputs = tokenizer.encode(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_length=256, top_k=1)

I don't know the reason, but it did work for me, anyone knows the answer?

[Stanislas0]

As the title says, on the same graphics card (3090), CodeGeeX2-6b is much slower than ChatGLM-6b. According to the official demo, I would like to know if there are any tricks.

It should be the same. The inference time also depends on the length of output tokens.