OpenBMB / BMTrain

Efficient Training (including pre-training and fine-tuning) for Big Models
Apache License 2.0
560 stars 77 forks source link

TypeError: expected string or bytes-like object #125

Closed dage0127 closed 1 year ago

dage0127 commented 1 year ago

编译时遇到问题: TypeError: expected string or bytes-like object [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure

OS:ubuntu18.04 gcc/g++:version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~18.04)

运行输出: Collecting bmtrain Using cached bmtrain-0.2.2.tar.gz (58 kB) Preparing metadata (setup.py) ... done Requirement already satisfied: numpy in /home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages (from bmtrain) (1.23.5) Building wheels for collected packages: bmtrain Building wheel for bmtrain (setup.py) ... error error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> [102 lines of output] No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda' running bdist_wheel running build running build_py creating build creating build/lib.linux-x86_64-cpython-310 creating build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/debug.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/init.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/layer.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/global_var.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/utils.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/checkpointing.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/parameter.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/store.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/block_layer.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/param_init.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/pipe_layer.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/init.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/wrapper.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/synchronize.py -> build/lib.linux-x86_64-cpython-310/bmtrain creating build/lib.linux-x86_64-cpython-310/bmtrain/nccl copying bmtrain/nccl/init.py -> build/lib.linux-x86_64-cpython-310/bmtrain/nccl copying bmtrain/nccl/enums.py -> build/lib.linux-x86_64-cpython-310/bmtrain/nccl creating build/lib.linux-x86_64-cpython-310/bmtrain/optim copying bmtrain/optim/init.py -> build/lib.linux-x86_64-cpython-310/bmtrain/optim copying bmtrain/optim/adam.py -> build/lib.linux-x86_64-cpython-310/bmtrain/optim copying bmtrain/optim/optim_manager.py -> build/lib.linux-x86_64-cpython-310/bmtrain/optim copying bmtrain/optim/adam_offload.py -> build/lib.linux-x86_64-cpython-310/bmtrain/optim creating build/lib.linux-x86_64-cpython-310/bmtrain/benchmark copying bmtrain/benchmark/init.py -> build/lib.linux-x86_64-cpython-310/bmtrain/benchmark copying bmtrain/benchmark/shape.py -> build/lib.linux-x86_64-cpython-310/bmtrain/benchmark copying bmtrain/benchmark/all_gather.py -> build/lib.linux-x86_64-cpython-310/bmtrain/benchmark copying bmtrain/benchmark/send_recv.py -> build/lib.linux-x86_64-cpython-310/bmtrain/benchmark copying bmtrain/benchmark/utils.py -> build/lib.linux-x86_64-cpython-310/bmtrain/benchmark copying bmtrain/benchmark/reduce_scatter.py -> build/lib.linux-x86_64-cpython-310/bmtrain/benchmark creating build/lib.linux-x86_64-cpython-310/bmtrain/inspect copying bmtrain/inspect/init.py -> build/lib.linux-x86_64-cpython-310/bmtrain/inspect copying bmtrain/inspect/format.py -> build/lib.linux-x86_64-cpython-310/bmtrain/inspect copying bmtrain/inspect/model.py -> build/lib.linux-x86_64-cpython-310/bmtrain/inspect copying bmtrain/inspect/tensor.py -> build/lib.linux-x86_64-cpython-310/bmtrain/inspect creating build/lib.linux-x86_64-cpython-310/bmtrain/loss copying bmtrain/loss/init.py -> build/lib.linux-x86_64-cpython-310/bmtrain/loss copying bmtrain/loss/cross_entropy.py -> build/lib.linux-x86_64-cpython-310/bmtrain/loss creating build/lib.linux-x86_64-cpython-310/bmtrain/lr_scheduler copying bmtrain/lr_scheduler/no_decay.py -> build/lib.linux-x86_64-cpython-310/bmtrain/lr_scheduler copying bmtrain/lr_scheduler/exponential.py -> build/lib.linux-x86_64-cpython-310/bmtrain/lr_scheduler copying bmtrain/lr_scheduler/init.py -> build/lib.linux-x86_64-cpython-310/bmtrain/lr_scheduler copying bmtrain/lr_scheduler/noam.py -> build/lib.linux-x86_64-cpython-310/bmtrain/lr_scheduler copying bmtrain/lr_scheduler/cosine.py -> build/lib.linux-x86_64-cpython-310/bmtrain/lr_scheduler copying bmtrain/lr_scheduler/warmup.py -> build/lib.linux-x86_64-cpython-310/bmtrain/lr_scheduler copying bmtrain/lr_scheduler/linear.py -> build/lib.linux-x86_64-cpython-310/bmtrain/lr_scheduler creating build/lib.linux-x86_64-cpython-310/bmtrain/distributed copying bmtrain/distributed/ops.py -> build/lib.linux-x86_64-cpython-310/bmtrain/distributed copying bmtrain/distributed/init.py -> build/lib.linux-x86_64-cpython-310/bmtrain/distributed running buildext Traceback (most recent call last): File "", line 2, in File "", line 34, in File "/tmp/pip-install-gdxukuv/bmtrain_71c9ac164edf4b6b83c5b8a68b907ba3/setup.py", line 74, in setup( File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/init.py", line 87, in setup return distutils.core.setup(**attrs) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup return run_commands(dist) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands dist.run_commands() File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands self.run_command(cmd) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command super().run_command(command) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 325, in run self.run_command("build") File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command self.distribution.run_command(command) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command super().run_command(command) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 132, in run self.run_command(cmd_name) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command self.distribution.run_command(command) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command super().run_command(command) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 84, in run _build_ext.run(self) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run self.build_extensions() File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 436, in build_extensions self._check_cuda_version(compiler_name, compiler_version) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 813, in _check_cuda_version torch_cuda_version = packaging.version.parse(torch.version.cuda) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/pkg_resources/_vendor/packaging/version.py", line 49, in parse return Version(version) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/pkg_resources/_vendor/packaging/version.py", line 264, in init match = self._regex.search(version) TypeError: expected string or bytes-like object [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for bmtrain Running setup.py clean for bmtrain Failed to build bmtrain Installing collected packages: bmtrain Running setup.py install for bmtrain ... error error: subprocess-exited-with-error

× Running setup.py install for bmtrain did not run successfully. │ exit code: 1 ╰─> [106 lines of output] No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda' running install /home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( running build running build_py creating build creating build/lib.linux-x86_64-cpython-310 creating build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/debug.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/init.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/layer.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/global_var.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/utils.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/checkpointing.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/parameter.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/store.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/block_layer.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/param_init.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/pipe_layer.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/init.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/wrapper.py -> build/lib.linux-x86_64-cpython-310/bmtrain copying bmtrain/synchronize.py -> build/lib.linux-x86_64-cpython-310/bmtrain creating build/lib.linux-x86_64-cpython-310/bmtrain/nccl copying bmtrain/nccl/init.py -> build/lib.linux-x86_64-cpython-310/bmtrain/nccl copying bmtrain/nccl/enums.py -> build/lib.linux-x86_64-cpython-310/bmtrain/nccl creating build/lib.linux-x86_64-cpython-310/bmtrain/optim copying bmtrain/optim/init.py -> build/lib.linux-x86_64-cpython-310/bmtrain/optim copying bmtrain/optim/adam.py -> build/lib.linux-x86_64-cpython-310/bmtrain/optim copying bmtrain/optim/optim_manager.py -> build/lib.linux-x86_64-cpython-310/bmtrain/optim copying bmtrain/optim/adam_offload.py -> build/lib.linux-x86_64-cpython-310/bmtrain/optim creating build/lib.linux-x86_64-cpython-310/bmtrain/benchmark copying bmtrain/benchmark/init.py -> build/lib.linux-x86_64-cpython-310/bmtrain/benchmark copying bmtrain/benchmark/shape.py -> build/lib.linux-x86_64-cpython-310/bmtrain/benchmark copying bmtrain/benchmark/all_gather.py -> build/lib.linux-x86_64-cpython-310/bmtrain/benchmark copying bmtrain/benchmark/send_recv.py -> build/lib.linux-x86_64-cpython-310/bmtrain/benchmark copying bmtrain/benchmark/utils.py -> build/lib.linux-x86_64-cpython-310/bmtrain/benchmark copying bmtrain/benchmark/reduce_scatter.py -> build/lib.linux-x86_64-cpython-310/bmtrain/benchmark creating build/lib.linux-x86_64-cpython-310/bmtrain/inspect copying bmtrain/inspect/init.py -> build/lib.linux-x86_64-cpython-310/bmtrain/inspect copying bmtrain/inspect/format.py -> build/lib.linux-x86_64-cpython-310/bmtrain/inspect copying bmtrain/inspect/model.py -> build/lib.linux-x86_64-cpython-310/bmtrain/inspect copying bmtrain/inspect/tensor.py -> build/lib.linux-x86_64-cpython-310/bmtrain/inspect creating build/lib.linux-x86_64-cpython-310/bmtrain/loss copying bmtrain/loss/init.py -> build/lib.linux-x86_64-cpython-310/bmtrain/loss copying bmtrain/loss/cross_entropy.py -> build/lib.linux-x86_64-cpython-310/bmtrain/loss creating build/lib.linux-x86_64-cpython-310/bmtrain/lr_scheduler copying bmtrain/lr_scheduler/no_decay.py -> build/lib.linux-x86_64-cpython-310/bmtrain/lr_scheduler copying bmtrain/lr_scheduler/exponential.py -> build/lib.linux-x86_64-cpython-310/bmtrain/lr_scheduler copying bmtrain/lr_scheduler/init.py -> build/lib.linux-x86_64-cpython-310/bmtrain/lr_scheduler copying bmtrain/lr_scheduler/noam.py -> build/lib.linux-x86_64-cpython-310/bmtrain/lr_scheduler copying bmtrain/lr_scheduler/cosine.py -> build/lib.linux-x86_64-cpython-310/bmtrain/lr_scheduler copying bmtrain/lr_scheduler/warmup.py -> build/lib.linux-x86_64-cpython-310/bmtrain/lr_scheduler copying bmtrain/lr_scheduler/linear.py -> build/lib.linux-x86_64-cpython-310/bmtrain/lr_scheduler creating build/lib.linux-x86_64-cpython-310/bmtrain/distributed copying bmtrain/distributed/ops.py -> build/lib.linux-x86_64-cpython-310/bmtrain/distributed copying bmtrain/distributed/init.py -> build/lib.linux-x86_64-cpython-310/bmtrain/distributed running buildext Traceback (most recent call last): File "", line 2, in File "", line 34, in File "/tmp/pip-install-gdxukuv/bmtrain_71c9ac164edf4b6b83c5b8a68b907ba3/setup.py", line 74, in setup( File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/init.py", line 87, in setup return distutils.core.setup(**attrs) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup return run_commands(dist) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands dist.run_commands() File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands self.run_command(cmd) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command super().run_command(command) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/command/install.py", line 68, in run return orig.install.run(self) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/_distutils/command/install.py", line 698, in run self.run_command('build') File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command self.distribution.run_command(command) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command super().run_command(command) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 132, in run self.run_command(cmd_name) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command self.distribution.run_command(command) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command super().run_command(command) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 84, in run _build_ext.run(self) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run self.build_extensions() File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 436, in build_extensions self._check_cuda_version(compiler_name, compiler_version) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 813, in _check_cuda_version torch_cuda_version = packaging.version.parse(torch.version.cuda) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/pkg_resources/_vendor/packaging/version.py", line 49, in parse return Version(version) File "/home/haitaiwork/llm/anaconda3/lib/python3.10/site-packages/pkg_resources/_vendor/packaging/version.py", line 264, in init match = self._regex.search(version) TypeError: expected string or bytes-like object [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure

× Encountered error while trying to install package. ╰─> bmtrain

note: This is an issue with the package mentioned above, not pip. hint: See above for output from the failure.

Achazwl commented 1 year ago

It seems that your Python env not working. See https://stackoverflow.com/questions/49060525/install-jupyterlab-in-pip3-throws-typeerror-expected-string-or-bytes-like-obje