Closed zhang-ge-hao closed 1 year ago
@wsxiaoys
Hi, thank you for providing such a great script.
Really hope that you could try using the script to transform a model that supports tensor parallelism.
Hi @AkiyamaYummy,
Unfortunately, the conversion script is maintained solely for Tabby's use case (a single GPU). Therefore, it is unlikely that fixing the tensor parallelism use case for us will be a priority. Maybe you could compare it with the GPT-J script (which works with tensor parallelism) and debug it yourself.
I try to use the script in this repo to convert gptneox model from huggingface model file to fastertransformer model file.
It worked when I converted files for single-GPU inference. However, when I converted a 2-GPU version of the fastertransformer model file that should work with tensor parallel, it generated nonsensical results.
Model: https://huggingface.co/TabbyML/NeoX-70M
Convert command:
1-gpu FT model result:
2-gpu FT model resut: