是否支持多 GPU。 - Githubissues

online2311 commented 1 year ago

单卡显存容量不够，一样可以支持多 GPU 把模型分摊到多 GPU 显存。

thomas-yanxin commented 1 year ago

今天会把代码放进去。但是我没有多卡的环境，所以不太方便测试。如果您方便，到时候能不能麻烦您帮忙测试一下？

thomas-yanxin commented 1 year ago

@online2311 代码已提交，您可以clone最新的代码进行尝试，若测试成功，能否在这个issue下反馈一下，感谢！

online2311 commented 1 year ago

今天会把代码放进去。但是我没有多卡的环境，所以不太方便测试。如果您方便，到时候能不能麻烦您帮忙测试一下？

好的，我稍后测试。测试完进行反馈。

thomas-yanxin commented 1 year ago

今天会把代码放进去。但是我没有多卡的环境，所以不太方便测试。如果您方便，到时候能不能麻烦您帮忙测试一下？

好的，我稍后测试。测试完进行反馈。

十分感谢！

online2311 commented 1 year ago

可以的，现在自动适配了两个 GPU。启动方式：docker run -it --rm --runtime=nvidia --gpus all --network host -v /home/nodecloud/ptuning/chatglm-6b:/data/chatglm-6b langchain-chatglm-webui bash

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla P100-PCIE-16GB            On | 00000000:04:00.0 Off |                    0 |
| N/A   44C    P0               32W / 250W|   8356MiB / 16384MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  Tesla P100-PCIE-16GB            On | 00000000:42:00.0 Off |                    0 |
| N/A   39C    P0               34W / 250W|   6470MiB / 16384MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A    436720      C   python                                     1126MiB |
|    0   N/A  N/A    443441      C   python3                                    7228MiB |
|    1   N/A  N/A    443441      C   python3                                    6468MiB |
+---------------------------------------------------------------------------------------+

root@nodecloud-ai-server:/build# python3 app.py 
Running on local URL:  http://0.0.0.0:7861

To create a public link, set `share=True` in `launch()`.
/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:27<00:00,  3.49s/it]
The dtype of attention mask (torch.int64) is not bool
{'query': '你好', 'result': '你好！请问有什么我可以帮助你的吗？', 'source_documents': [Document(page_content='教训', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='提出', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='意义', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='意义', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='内容：', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='武昌起义', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'})]}

online2311 commented 1 year ago

我在考虑是否替换为 LLaMA 13B的模型，是否会更好。

online2311 commented 1 year ago

缺陷：每次问答都需要重新加载模型，导致每次回答时间超长。

/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:27<00:00,  3.49s/it]
The dtype of attention mask (torch.int64) is not bool
{'query': '你好', 'result': '你好！请问有什么我可以帮助你的吗？', 'source_documents': [Document(page_content='教训', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='提出', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='意义', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='意义', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='内容：', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='武昌起义', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'})]}
/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:20<00:00,  2.58s/it]
{'query': '广州起义是发起的。', 'result': '根据已知信息无法回答该问题，因为没有提供足够的相关信息。广州起义是1924年10月10日发起的，由中国共产党领导的革命力量发起的。', 'source_documents': [Document(page_content='要不要社会革命。', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='26.太平天国农民起义爆发的时间是（A）', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='南昌起义的意义：', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='62.1925年全国范围大革命风暴兴起的标志是(D)。', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='72.简答题：秋收起义的特点。', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='2、组建中华革命党。', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'})]}
/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:15<00:00,  1.92s/it]
{'query': '四海志是谁编写的？', 'result': '根据已知信息，没有提供足够的相关信息，无法回答该问题。', 'source_documents': [Document(page_content='要不要社会革命。', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='要不要推翻帝制，实行共和。', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='D、要不要废科举，兴学堂', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='196．1979年元旦，全国人大常委会发表的重要文献是《告台湾同胞书》。', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='A.《驳康有为论革命书》', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='关于革命与改良的辩论', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'})]}
/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx

thomas-yanxin commented 1 year ago

我在考虑是否替换为 LLaMA 13B的模型，是否会更好。

LLaMA 13B的模型对于中文似乎不够好，社区反馈Vicuna可能综合效果会更好一些。

最近我也在调研各种LLM，也希望能够支持更多的LLM，如果您愿意，我们可以一起来做这件事情。

thomas-yanxin commented 1 year ago

缺陷：每次问答都需要重新加载模型，导致每次回答时间超长。

/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:27<00:00,  3.49s/it]
The dtype of attention mask (torch.int64) is not bool
{'query': '你好', 'result': '你好！请问有什么我可以帮助你的吗？', 'source_documents': [Document(page_content='教训', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='提出', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='意义', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='意义', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='内容：', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='武昌起义', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'})]}
/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:20<00:00,  2.58s/it]
{'query': '广州起义是发起的。', 'result': '根据已知信息无法回答该问题，因为没有提供足够的相关信息。广州起义是1924年10月10日发起的，由中国共产党领导的革命力量发起的。', 'source_documents': [Document(page_content='要不要社会革命。', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='26.太平天国农民起义爆发的时间是（A）', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='南昌起义的意义：', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='62.1925年全国范围大革命风暴兴起的标志是(D)。', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='72.简答题：秋收起义的特点。', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='2、组建中华革命党。', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'})]}
/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:15<00:00,  1.92s/it]
{'query': '四海志是谁编写的？', 'result': '根据已知信息，没有提供足够的相关信息，无法回答该问题。', 'source_documents': [Document(page_content='要不要社会革命。', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='要不要推翻帝制，实行共和。', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='D、要不要废科举，兴学堂', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='196．1979年元旦，全国人大常委会发表的重要文献是《告台湾同胞书》。', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='A.《驳康有为论革命书》', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='关于革命与改良的辩论', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'})]}
/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx

问题收到！谢谢反馈！后续会对此进行优化！

online2311 commented 1 year ago

Embedding model 无法切换，切换回报错。

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/gradio/routes.py", line 401, in run_predict
    output = await app.get_blocks().process_api(
  File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1302, in process_api
    result = await self.call_function(
  File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1025, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/build/app.py", line 105, in predict
    vector_store = init_knowledge_vector_store(embedding_model, file_obj.name)
  File "/build/app.py", line 40, in init_knowledge_vector_store
    model_name=embedding_model_dict[embedding_model], )
KeyError: 'text2vec-base'

online2311 commented 1 year ago

我在考虑是否替换为 LLaMA 13B的模型，是否会更好。

LLaMA 13B的模型对于中文似乎不够好，社区反馈Vicuna可能综合效果会更好一些。

最近我也在调研各种LLM，也希望能够支持更多的LLM，如果您愿意，我们可以一起来做这件事情。

我说的是 BELLE-LLaMA-13B-2M 感觉对中文理解更好一些相对于 ChartGLM

thomas-yanxin commented 1 year ago

我在考虑是否替换为 LLaMA 13B的模型，是否会更好。

LLaMA 13B的模型对于中文似乎不够好，社区反馈Vicuna可能综合效果会更好一些。最近我也在调研各种LLM，也希望能够支持更多的LLM，如果您愿意，我们可以一起来做这件事情。

我说的是 BELLE-LLaMA-13B-2M 感觉对中文理解更好一些相对于 ChatGLM

哈哈，这个模型我也有关注了

online2311 commented 1 year ago

Embedding model 无法切换，切换回报错。

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/gradio/routes.py", line 401, in run_predict
    output = await app.get_blocks().process_api(
  File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1302, in process_api
    result = await self.call_function(
  File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1025, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/build/app.py", line 105, in predict
    vector_store = init_knowledge_vector_store(embedding_model, file_obj.name)
  File "/build/app.py", line 40, in init_knowledge_vector_store
    model_name=embedding_model_dict[embedding_model], )
KeyError: 'text2vec-base'

这块好像写错了 text2vec 应该是 text2vec-base吧。

embedding_model_dict = {
    "ernie-tiny": "nghuyong/ernie-3.0-nano-zh",
    "ernie-base": "nghuyong/ernie-3.0-base-zh",
    "ernie-medium": "nghuyong/ernie-3.0-medium-zh",
    "ernie-xbase": "nghuyong/ernie-3.0-xbase-zh",
    "text2vec": "GanymedeNil/text2vec-large-chinese"
}

thomas-yanxin commented 1 year ago

Embedding model 无法切换，切换回报错。

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/gradio/routes.py", line 401, in run_predict
    output = await app.get_blocks().process_api(
  File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1302, in process_api
    result = await self.call_function(
  File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1025, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/build/app.py", line 105, in predict
    vector_store = init_knowledge_vector_store(embedding_model, file_obj.name)
  File "/build/app.py", line 40, in init_knowledge_vector_store
    model_name=embedding_model_dict[embedding_model], )
KeyError: 'text2vec-base'

这块好像写错了 text2vec 应该是 text2vec-base吧。

embedding_model_dict = {
    "ernie-tiny": "nghuyong/ernie-3.0-nano-zh",
    "ernie-base": "nghuyong/ernie-3.0-base-zh",
    "ernie-medium": "nghuyong/ernie-3.0-medium-zh",
    "ernie-xbase": "nghuyong/ernie-3.0-xbase-zh",
    "text2vec": "GanymedeNil/text2vec-large-chinese"
}

如果您方便的话，可以提个PR！~

online2311 commented 1 year ago

@thomas-yanxin 修改已提交PR.

thomas-yanxin commented 1 year ago

可以的，现在自动适配了两个 GPU。启动方式：docker run -it --rm --runtime=nvidia --gpus all --network host -v /home/nodecloud/ptuning/chatglm-6b:/data/chatglm-6b langchain-chatglm-webui bash

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla P100-PCIE-16GB            On | 00000000:04:00.0 Off |                    0 |
| N/A   44C    P0               32W / 250W|   8356MiB / 16384MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  Tesla P100-PCIE-16GB            On | 00000000:42:00.0 Off |                    0 |
| N/A   39C    P0               34W / 250W|   6470MiB / 16384MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A    436720      C   python                                     1126MiB |
|    0   N/A  N/A    443441      C   python3                                    7228MiB |
|    1   N/A  N/A    443441      C   python3                                    6468MiB |
+---------------------------------------------------------------------------------------+

root@nodecloud-ai-server:/build# python3 app.py 
Running on local URL:  http://0.0.0.0:7861

To create a public link, set `share=True` in `launch()`.
/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:27<00:00,  3.49s/it]
The dtype of attention mask (torch.int64) is not bool
{'query': '你好', 'result': '你好！请问有什么我可以帮助你的吗？', 'source_documents': [Document(page_content='教训', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='提出', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='意义', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='意义', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='内容：', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='武昌起义', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'})]}

您好！dev分支更改了一下多卡推理的代码，如果您方便，能否再帮忙测试一下？

thomas-yanxin commented 1 year ago

我在考虑是否替换为 LLaMA 13B的模型，是否会更好。

LLaMA 13B的模型对于中文似乎不够好，社区反馈Vicuna可能综合效果会更好一些。最近我也在调研各种LLM，也希望能够支持更多的LLM，如果您愿意，我们可以一起来做这件事情。

我说的是 BELLE-LLaMA-13B-2M 感觉对中文理解更好一些相对于 ChartGLM

dev分支支持了Vicuna，可以一起看一下效果，另外我测了一下Belle-LLaMA-7B-2M，推理速度感人，是否有必要集成进来？能否讨论一下~

thomas-yanxin commented 1 year ago

缺陷：每次问答都需要重新加载模型，导致每次回答时间超长。

/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:27<00:00,  3.49s/it]
The dtype of attention mask (torch.int64) is not bool
{'query': '你好', 'result': '你好！请问有什么我可以帮助你的吗？', 'source_documents': [Document(page_content='教训', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='提出', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='意义', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='意义', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='内容：', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='武昌起义', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'})]}
/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:20<00:00,  2.58s/it]
{'query': '广州起义是发起的。', 'result': '根据已知信息无法回答该问题，因为没有提供足够的相关信息。广州起义是1924年10月10日发起的，由中国共产党领导的革命力量发起的。', 'source_documents': [Document(page_content='要不要社会革命。', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='26.太平天国农民起义爆发的时间是（A）', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='南昌起义的意义：', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='62.1925年全国范围大革命风暴兴起的标志是(D)。', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='72.简答题：秋收起义的特点。', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='2、组建中华革命党。', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'})]}
/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
No sentence-transformers model found with name /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh. Creating a new one with MEAN pooling.
Some weights of the model checkpoint at /root/.cache/torch/sentence_transformers/nghuyong_ernie-3.0-base-zh were not used when initializing ErnieModel: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']
- This IS expected if you are initializing ErnieModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ErnieModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a configuration with custom code to ensure no malicious code has been contributed in a newer revision.
Explicitly passing a `revision` is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:15<00:00,  1.92s/it]
{'query': '四海志是谁编写的？', 'result': '根据已知信息，没有提供足够的相关信息，无法回答该问题。', 'source_documents': [Document(page_content='要不要社会革命。', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='要不要推翻帝制，实行共和。', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='D、要不要废科举，兴学堂', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='196．1979年元旦，全国人大常委会发表的重要文献是《告台湾同胞书》。', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='A.《驳康有为论革命书》', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'}), Document(page_content='关于革命与改良的辩论', metadata={'source': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'filename': '/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx', 'category': 'Title'})]}
/tmp/4e9a105bfba5239426c8014f68ef8a06c72acc64/中国近现代史纲要复习资料.docx

这个想到了一些办法，但会损失交互的体验。看是否有应当追求速度而损失交互，可以讨论一下~感谢！

online2311 commented 1 year ago

我在考虑是否替换为 LLaMA 13B的模型，是否会更好。

LLaMA 13B的模型对于中文似乎不够好，社区反馈Vicuna可能综合效果会更好一些。最近我也在调研各种LLM，也希望能够支持更多的LLM，如果您愿意，我们可以一起来做这件事情。

我说的是 BELLE-LLaMA-13B-2M 感觉对中文理解更好一些相对于 ChartGLM

dev分支支持了Vicuna，可以一起看一下效果，另外我测了一下Belle-LLaMA-7B-2M，推理速度感人，是否有必要集成进来？能否讨论一下~

我用其他项目去跑 Belle-LLaMA-7B-2M 与 Belle-LLaMA-13B-2M 速度很快。而且 LLaMA 很长一段时间还是主流。

thomas-yanxin commented 1 year ago

我在考虑是否替换为 LLaMA 13B的模型，是否会更好。

LLaMA 13B的模型对于中文似乎不够好，社区反馈Vicuna可能综合效果会更好一些。最近我也在调研各种LLM，也希望能够支持更多的LLM，如果您愿意，我们可以一起来做这件事情。

我说的是 BELLE-LLaMA-13B-2M 感觉对中文理解更好一些相对于 ChartGLM

dev分支支持了Vicuna，可以一起看一下效果，另外我测了一下Belle-LLaMA-7B-2M，推理速度感人，是否有必要集成进来？能否讨论一下~

我用其他项目去跑 Belle-LLaMA-7B-2M 与 Belle-LLaMA-13B-2M 速度很快。而且 LLaMA 很长一段时间还是主流。

哦对，新版本已经集成了Belle-LLaMA-7B-2M，我改了一下模型加载那里，速度优化了，您有空可以试试看。但是Belle和Vicuna都没有加history，这里您要是方便可以提个PR。

raymondino commented 1 year ago

你好！请问多卡推理那一块的code在哪里呀？我找了半天没找到 - 看了online2311的PR 没有找到4/19号的commit。我现在遇到的问题：我有4个GPU，但是用langchain做inference时候，GPU的load不是很balance，总是第一个GPU load多，最后一个GPU的memory 少 - 请问你知道怎么把每个GPU的load给balance一下吗？

X-D-Lab / LangChain-ChatGLM-Webui

是否支持多 GPU。 #4