InternLM / xtuner

An efficient, flexible and full-featured toolkit for fine-tuning LLM (InternLM2, Llama3, Phi3, Qwen, Mistral, ...)
https://xtuner.readthedocs.io/zh-cn/latest/
Apache License 2.0
3.94k stars 308 forks source link

docker容器里,pip install xtuner后,提示bash: xtuner: command not found #157

Closed chelovek21 closed 1 year ago

chelovek21 commented 1 year ago

docker容器里,按照如下方式安装了xtuner git clone https://github.com/InternLM/xtuner.git cd xtuner pip install -e '.[all]' 其它依赖环境都以配置好了, 在xtuner安装路径下,执行命令 xtuner list-cfg 提示错误: bash: xtuner: command not found 这是什么原因,还需要配置软连接之类的吗? pip list可以看到xtuner,本地安装成功了0.1.4版本。

chelovek21 commented 1 year ago

尝试直接用train.py脚本直接调用也会解析config文件错误。 执行 NPROC_PER_NODE=4 python xtuner/tools/train.py xtuner/configs/qwen/qwen_7b_chat/qwen_7b_chat_qlora_oasst1_e3.py

显示错误如下: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/mmengine/config/config.py", line 1477, in prettytext text, = FormatCode(text, style_config=yapf_style) File "/opt/conda/lib/python3.10/site-packages/yapf/yapflib/yapf_api.py", line 201, in FormatCode raise errors.YapfError(errors.FormatErrorMsg(e)) yapf.yapflib.errors.YapfError: :13:14: invalid syntax

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/mnt/LLM/workspace/internLM/xtuner/xtuner/tools/train.py", line 246, in main() File "/mnt/LLM/workspace/internLM/xtuner/xtuner/tools/train.py", line 235, in main runner = Runner.from_cfg(cfg) File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/runner.py", line 445, in from_cfg runner = cls( File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/runner.py", line 386, in init self._log_env(env_cfg) File "/opt/conda/lib/python3.10/site-packages/mmengine/runner/runner.py", line 2356, in _log_env self.logger.info(f'Config:\n{self.cfg.pretty_text}') File "/opt/conda/lib/python3.10/site-packages/mmengine/config/config.py", line 1482, in pretty_text raise SyntaxError('Failed to format the config file, please ' SyntaxError: Failed to format the config file, please check the syntax of:

LZHgrla commented 1 year ago

https://github.com/open-mmlab/mmengine/issues/1364

这个应该是与您遇到了类似的问题,可以从源码安装mmengine(相关pr尚未发版),或安装低版本yapf解决

pip uninstall mmengine; pip install git+https://github.com/open-mmlab/mmengine.git
pip install yapf==0.40.1
LZHgrla commented 1 year ago

docker容器里,按照如下方式安装了xtuner git clone https://github.com/InternLM/xtuner.git cd xtuner pip install -e '.[all]' 其它依赖环境都以配置好了, 在xtuner安装路径下,执行命令 xtuner list-cfg 提示错误: bash: xtuner: command not found 这是什么原因,还需要配置软连接之类的吗? pip list可以看到xtuner,本地安装成功了0.1.4版本。

xtuner 会安装到pip、python对应的那个bin文件夹(例如~/anaconda3/envs/my_env/bin/),可以查看一下该文件夹下是否有对应的程序

chelovek21 commented 1 year ago

open-mmlab/mmengine#1364

这个应该是与您遇到了类似的问题,可以从源码安装mmengine(相关pr尚未发版),或安装低版本yapf解决

pip uninstall mmengine; pip install git+https://github.com/open-mmlab/mmengine.git
pip install yapf==0.40.1

是的,按照这种方式就可以解决了,感谢您的及时回复。

chelovek21 commented 1 year ago

docker容器里,按照如下方式安装了xtuner git clone https://github.com/InternLM/xtuner.git cd xtuner pip install -e '.[all]' 其它依赖环境都以配置好了, 在xtuner安装路径下,执行命令 xtuner list-cfg 提示错误: bash: xtuner: command not found 这是什么原因,还需要配置软连接之类的吗? pip list可以看到xtuner,本地安装成功了0.1.4版本。

xtuner 会安装到pip、python对应的那个bin文件夹(例如~/anaconda3/envs/my_env/bin/),可以查看一下该文件夹下是否有对应的程序

确实,输入绝对路径即可启动xtuner命令了,我建个软链到系统bin即可使用,谢谢