search

DaoMingze / zhukebot

我的nonebot机器人
MIT License
14 stars 1 forks source link

[BUG]Windows系统环境中的兼容性错误 #4

Closed wling-art closed 1 year ago

wling-art commented 1 year ago

日志，windows不能用torch.compile

04-30 05:38:39 [ERROR] nonebot | Failed to import "nonebot_plugin_chatglm"
Traceback (most recent call last):
  File "e:\Users\wling\Desktop\code\U1\bot.py", line 12, in <module>
    nonebot.load_from_toml("pyproject.toml")
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\plugin\load.py", line 125, in load_from_toml
    return load_all_plugins(plugins, plugin_dirs)
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\plugin\load.py", line 62, in load_all_plugins
    return manager.load_all_plugins()
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\plugin\manager.py", line 175, in load_all_plugins
    return set(
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\plugin\manager.py", line 176, in <genexpr>
    filter(None, (self.load_plugin(name) for name in self.available_plugins))
> File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\plugin\manager.py", line 141, in load_plugin
    module = importlib.import_module(name)
  File "C:\Users\wling\AppData\Local\Programs\Python\Python311\Lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\plugin\manager.py", line 240, in exec_module
    super().exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot_plugin_chatglm\__init__.py", line 3, in <module>
    from . import chat, config
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot_plugin_chatglm\chat.py", line 18, in <module>
    from .config import *
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot_plugin_chatglm\config.py", line 86, in <module>
    model = compile(model).eval()
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\torch\__init__.py", line 1441, in compile
    return torch._dynamo.optimize(backend=backend, nopython=fullgraph, dynamic=dynamic, disable=disable)(model)
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\torch\_dynamo\eval_frame.py", line 413, in optimize
    check_if_dynamo_supported()
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\torch\_dynamo\eval_frame.py", line 375, in check_if_dynamo_supported
    raise RuntimeError("Windows not yet supported for torch.compile")
RuntimeError: Windows not yet supported for torch.compile
wling-art commented 1 year ago

环境：cuda11.8 pytorch2.0 windows 11

DaoMingze commented 1 year ago

好像是pytorch就有这个问题，PyTorch 2.0 not working on Windows #90768

我看看怎么解决，不行就把torch.complie(model)这行删了，我是没感觉加速效果，只是想着多一行提速不亏。

wling-art commented 1 year ago

然后然报错了一个😢

04-30 14:26:39 [ERROR] nonebot | Failed to import "nonebot_plugin_chatglm"
Traceback (most recent call last):
  File "e:\Users\wling\Desktop\code\U1\bot.py", line 9, in <module>
    nonebot.load_from_toml("pyproject.toml")
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\plugin\load.py", line 125, in load_from_toml
    return load_all_plugins(plugins, plugin_dirs)
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\plugin\load.py", line 62, in load_all_plugins
    return manager.load_all_plugins()
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\plugin\manager.py", line 175, in load_all_plugins
    return set(
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\plugin\manager.py", line 176, in <genexpr>
    filter(None, (self.load_plugin(name) for name in self.available_plugins))
> File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\plugin\manager.py", line 141, in load_plugin
    module = importlib.import_module(name)
  File "C:\Users\wling\AppData\Local\Programs\Python\Python311\Lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\plugin\manager.py", line 240, in exec_module
    super().exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot_plugin_chatglm\__init__.py", line 3, in <module>
    from . import chat, config
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot_plugin_chatglm\chat.py", line 18, in <module>
    from .config import *
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot_plugin_chatglm\config.py", line 66, in <module>
    tokenizer = AutoTokenizer.from_pretrained(model_path,
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\transformers\models\auto\tokenization_auto.py", line 686, in from_pretrained
    tokenizer_class = get_class_from_dynamic_module(
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\transformers\dynamic_module_utils.py", line 388, in get_class_from_dynamic_module
    return get_class_in_module(class_name, final_module.replace(".py", ""))
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\transformers\dynamic_module_utils.py", line 157, in get_class_in_module
    module = importlib.import_module(module_path)
  File "C:\Users\wling\AppData\Local\Programs\Python\Python311\Lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'transformers_modules.'
wling-art commented 1 year ago

transformers-4.28.1

DaoMingze commented 1 year ago

我的锅。我以为我是分开写的，结果写到一起了。

config.py第86行model= complie(model).eval()改成model=model.eval()

DaoMingze commented 1 year ago

我的锅。我以为我是分开写的，结果写到一起了。

config.py第86行model= complie(model).eval()改成model=model.eval()

@wlingzhenyu 抱歉，没说清楚（也没按规范分开写）

wling-art commented 1 year ago

还是会出现报错😶‍🌫️，貌似还是没解决

正从./data/chatglm/model加载模型
04-30 14:51:13 [ERROR] nonebot | Failed to import "nonebot_plugin_chatglm"
Traceback (most recent call last):
  File "e:\Users\wling\Desktop\code\U1\bot.py", line 9, in <module>
    nonebot.load_from_toml("pyproject.toml")
  File "E:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\plugin\load.py", line 125, in load_from_toml
    return load_all_plugins(plugins, plugin_dirs)
  File "E:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\plugin\load.py", line 62, in load_all_plugins
    return manager.load_all_plugins()
  File "E:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\plugin\manager.py", line 175, in load_all_plugins
    return set(
  File "E:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\plugin\manager.py", line 176, in <genexpr>
    filter(None, (self.load_plugin(name) for name in self.available_plugins))
> File "E:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\plugin\manager.py", line 141, in load_plugin
    module = importlib.import_module(name)
  File "C:\Users\wling\AppData\Local\Programs\Python\Python311\Lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "E:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\plugin\manager.py", line 240, in exec_module
    super().exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "E:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot_plugin_chatglm\__init__.py", line 3, in <module>
    from . import chat, config
  File "E:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot_plugin_chatglm\chat.py", line 18, in <module>
    from .config import *
  File "E:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot_plugin_chatglm\config.py", line 66, in <module>
    tokenizer = AutoTokenizer.from_pretrained(
  File "E:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\transformers\models\auto\tokenization_auto.py", line 686, in from_pretrained
    tokenizer_class = get_class_from_dynamic_module(
  File "E:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\transformers\dynamic_module_utils.py", line 388, in get_class_from_dynamic_module
    return get_class_in_module(class_name, final_module.replace(".py", ""))
  File "E:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\transformers\dynamic_module_utils.py", line 157, in get_class_in_module
    module = importlib.import_module(module_path)
  File "C:\Users\wling\AppData\Local\Programs\Python\Python311\Lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'transformers_modules.'
wling-art commented 1 year ago

有可能是我的transformers有版本问题吗

wling-art commented 1 year ago

transformers 4.28.1 cuda 11.8 pytorch 2.0

DaoMingze commented 1 year ago

有可能是我的transformers有版本问题吗

我的也更新到了4.28.1但是没问题。所以应该不是transformers的问题。我仔细看了一下日志，这里显示是autotokenizer加载的问题

File "E:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot_plugin_chatglm\config.py", line 66, in <module>
    tokenizer = AutoTokenizer.from_pretrained(

我仔细看看，你可以加我qq，1352488443交流一下详细情况。

DaoMingze commented 1 year ago

我想应该是这样一个情况：我准备的默认安装路径是Linux格式的，自动下载并转移保存后，再次加载时，autotokenizer找到的路径（modelpath）出错了。你检查一下./data/chatglm/model这个路径下是否有下载好的模型文件，我看日志好像是第二次就有了，然后复制一下绝对路径，写到.env里（.env.prod或者.env.dev）

大致是chaglm_model="路径"。

wling-art commented 1 year ago

我想应该是这样一个情况：我准备的默认安装路径是Linux格式的，自动下载并转移保存后，再次加载时，autotokenizer找到的路径（modelpath）出错了。你检查一下./data/chatglm/model这个路径下是否有下载好的模型文件，我看日志好像是第二次就有了，然后复制一下绝对路径，写到.env里（.env.prod或者.env.dev）

大致是chaglm_model="路径"。

chatglm_model指定后成功了，但是当我私聊向他发送hi时有发生了错误😢

05-01 00:03:04 [SUCCESS] nonebot | OneBot V11 1184441051 | [message.private.friend]: Message -465143847 from 1990248284 'hi'
05-01 00:03:04 [DEBUG] nonebot | Checking for matchers in priority 1...
05-01 00:03:04 [DEBUG] nonebot | Checking for matchers in priority 2...
05-01 00:03:04 [DEBUG] nonebot | Checking for matchers in priority 12...
05-01 00:03:04 [DEBUG] nonebot | Checking for matchers in priority 15...
05-01 00:03:04 [DEBUG] nonebot | Checking for matchers in priority 30...
05-01 00:03:04 [DEBUG] nonebot | Checking for matchers in priority 40...
05-01 00:03:04 [DEBUG] nonebot | Checking for matchers in priority 50...
05-01 00:03:04 [INFO] nonebot | Event will be handled by Matcher(type='message', module=nonebot_plugin_chatglm.chat)
05-01 00:03:04 [DEBUG] nonebot | Running Matcher(type='message', module=nonebot_plugin_chatglm.chat)
05-01 00:03:04 [DEBUG] nonebot | Running handler Dependent(call=chat)
False
[]
The dtype of attention mask (torch.int64) is not bool
05-01 00:03:04 [ERROR] nonebot_plugin_chatglm | 生成失败
Traceback (most recent call last):
  File "e:\Users\wling\Desktop\code\U1\bot.py", line 12, in <module>
    nonebot.run()
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\__init__.py", line 309, in run
    get_driver().run(*args, **kwargs)
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\drivers\fastapi.py", line 198, in run
    uvicorn.run(
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\uvicorn\main.py", line 578, in run
    server.run()
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\uvicorn\server.py", line 61, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "C:\Users\wling\AppData\Local\Programs\Python\Python311\Lib\asyncio\runners.py", line 190, in run
    return runner.run(main)
  File "C:\Users\wling\AppData\Local\Programs\Python\Python311\Lib\asyncio\runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File "C:\Users\wling\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 640, in run_until_complete
    self.run_forever()
  File "C:\Users\wling\AppData\Local\Programs\Python\Python311\Lib\asyncio\windows_events.py", line 321, in run_forever
    super().run_forever()
  File "C:\Users\wling\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 607, in run_forever
    self._run_once()
  File "C:\Users\wling\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 1922, in _run_once
    handle._run()
  File "C:\Users\wling\AppData\Local\Programs\Python\Python311\Lib\asyncio\events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\message.py", line 141, in _check_matcher
    await _run_matcher(Matcher, bot, event, state, stack, dependency_cache)
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\message.py", line 187, in _run_matcher
    await matcher.run(bot, event, state, stack, dependency_cache)
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\internal\matcher\matcher.py", line 753, in run
    await self.simple_run(bot, event, state, stack, dependency_cache)
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\internal\matcher\matcher.py", line 728, in simple_run
    await handler(
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot\dependencies\__init__.py", line 108, in __call__
    return await cast(Callable[..., Awaitable[R]], self.call)(**values)
> File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\nonebot_plugin_chatglm\chat.py", line 95, in chat
    response, new = model.chat(tokenizer, query, history=his)
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\wling/.cache\huggingface\modules\transformers_modules\model\modeling_chatglm.py", line 1253, in chat
    outputs = self.generate(**inputs, **gen_kwargs)
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\transformers\generation\utils.py", line 1485, in generate
    return self.sample(
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\transformers\generation\utils.py", line 2524, in sample
    outputs = self(
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\wling/.cache\huggingface\modules\transformers_modules\model\modeling_chatglm.py", line 1158, in forward
    transformer_outputs = self.transformer(
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\wling/.cache\huggingface\modules\transformers_modules\model\modeling_chatglm.py", line 905, in forward
    inputs_embeds = self.word_embeddings(input_ids)
  File "e:\Users\wling\Desktop\code\U1\.venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\wling/.cache\huggingface\modules\transformers_modules\model\quantization.py", line 382, in forward
    original_weight = extract_weight_to_float(weight=self.weight, scale_list=self.weight_scale, source_bit_width=self.weight_bit_width)
  File "C:\Users\wling/.cache\huggingface\modules\transformers_modules\model\quantization.py", line 260, in extract_weight_to_float
    func = cpu_kernels.int4WeightExtractionFloat
AttributeError: 'NoneType' object has no attribute 'int4WeightExtractionFloat'
05-01 00:03:04 [DEBUG] nonebot | OneBot V11 | Calling API send_msg
DaoMingze commented 1 year ago

你发一下chatglm的配置。最终报错结果是没有属性int4WeightExtractionFloat，可能是设置成了本地量化。

现在默认的模型是chatglm-6b-int4-qe，搭配的应该是chatglm_mode=cuda。

不过默认的cpu(没写chatglm_mode就是cpu运行)运行我确实没测试过，只测试过chatglm-6b模型的。等解决了你的问题，我再统一修复一下这些因为疏忽造成的bug:

  1. torch.complie做成根据系统环境生效
  2. 默认模型存放和使用路径根据系统环境生效
  3. 检查不同模型及策略的组合生效情况([cpu | cuda]×[chatglm-6b | chatglm-6b-int8 |chatglm-6b-int4 | chatglm-6b-int4-qe])
DaoMingze commented 1 year ago

纯cpu推理我是想用JittorLLMs的，问题只在于Jittor魔改了torch，我怕引入新的bug，且影响其他部署好了的项目(比如说stable diffusion)，所以没加。

今天有事出门，晚上回家看怎么尽量节省资源进行推理。

wling-art commented 1 year ago

我使用了chatglm-6b的模型，成功了，但出现了个问题，生成的过程中貌似会占用线程，导致其他功能出现问题

这个问题可以在下一个版本中解决咩

DaoMingze commented 1 year ago

我使用了chatglm-6b的模型，成功了，但出现了个问题，生成的过程中貌似会占用线程，导致其他功能出现问题

这个问题可以在下一个版本中解决咩

ok

DaoMingze commented 1 year ago

看ChatGLM-6B的官方说明，

如果需要在 cpu 上运行量化后的模型，还需要安装 gcc 与 openmp。多数 Linux 发行版默认已安装。对于 Windows ，可在安装 TDM-GCC 时勾选 openmp。 Windows 测试环境 gcc 版本为 TDM-GCC 10.3.0， Linux 为 gcc 11.3.0。在 MacOS 上请参考 Q1。

在Windows系统上用量化后的模型（int8、int4、int4-qe），要安装gcc。

DaoMingze commented 1 year ago

我使用了chatglm-6b的模型，成功了，但出现了个问题，生成的过程中貌似会占用线程，导致其他功能出现问题

这个问题可以在下一个版本中解决咩

由于之前的思路和写法是用await，所以它是等待返回回复才执行其他命令功能（即占用线程）。API的思路比较容易解决这个问题，是交给（post）另一个线程（API）然后监听等待回复。

你可以试试nonebot_plugin_ChatGLM6B来实现。

在CUDA推理模式下，较简短的输入大概等候也就10秒内，而CPU推理则30秒或几分钟。本地CPU部署我目前的思路是尽量用别人加速的来实现（比如说JittorLLMs，但这个魔改torch，怕出兼容性问题）