[REQUEST] Tensor-parallel support for Gemma2ForCausalLM

[Abdulhanan535]

Problem

Recently i saw the Tensor-parallel post, and i tried it for gemma2 27B but it didn't work i'm using 8.0bpw exl2 of it

Tensor-parallel is not supported for Gemma2ForCausalLM?

Solution

idk.

Alternatives

No response

Explanation

Tensor-parallel support for Gemma2ForCausalLM possible?

Examples

No response

Additional context

No response

Acknowledgements

• [X] I have looked for similar requests before submitting this one.
• [X] I understand that the developers have lives and my issue will be answered when possible.
• [X] I understand the developers of this program are human, and I will make my requests politely.

[turboderp]

Tensor-parallel is only implemented for Llama, Qwen2 and Mistral for the time being.

[bdashore3]

Closing as this is an exllamav2 issue. Please open an issue there instead of TabbyAPI.