anakinxc
commented
2 months ago hi @grueyg
please make sure you are using lld as linker.
Closed grueyg closed 2 months ago
anakinxc
commented
2 months ago hi @grueyg
please make sure you are using lld as linker.
grueyg
commented
2 months ago 谢谢 @anakinxc
在安装了lld，以及在.bazerlrc文件中添加
build:linux --linkopt=-fuse-ld=lld可以正确编译了
grueyg
commented
2 months ago @anakinxc，很抱歉，我的安装仍然存在问题。
我在一个新环境里以可编辑模式安装，执行pip install -e .后出现报错，主要原因似乎是ModuleNotFoundError: No module named 'numpy'
但是，我明明已经安装了numpy，并且在该环境下的其他代码里也可以正确import numpy，这是哪里出了问题？
我的安装方法如下
cd spu
pip install -r requirements.txt
pip install -r requirements-dev.txt
pip install -e .报错如下
(sf) userA@ubuntu:~/project/sf/spu$ pip install -e .
Obtaining file:///home/userA/project/sf/spu
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... done
Installing backend dependencies ... done
Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: spu
Building editable for spu (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building editable for spu (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [196 lines of output]
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
running editable_wheel
creating /tmp/pip-wheel-24l08imq/.tmp-5u5hkj06/spu.egg-info
writing /tmp/pip-wheel-24l08imq/.tmp-5u5hkj06/spu.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-wheel-24l08imq/.tmp-5u5hkj06/spu.egg-info/dependency_links.txt
writing requirements to /tmp/pip-wheel-24l08imq/.tmp-5u5hkj06/spu.egg-info/requires.txt
writing top-level names to /tmp/pip-wheel-24l08imq/.tmp-5u5hkj06/spu.egg-info/top_level.txt
writing manifest file '/tmp/pip-wheel-24l08imq/.tmp-5u5hkj06/spu.egg-info/SOURCES.txt'
reading manifest file '/tmp/pip-wheel-24l08imq/.tmp-5u5hkj06/spu.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file '/tmp/pip-wheel-24l08imq/.tmp-5u5hkj06/spu.egg-info/SOURCES.txt'
creating '/tmp/pip-wheel-24l08imq/.tmp-5u5hkj06/spu-0.9.3.dev20240829.dist-info'
creating /tmp/pip-wheel-24l08imq/.tmp-5u5hkj06/spu-0.9.3.dev20240829.dist-info/WHEEL
running build_py
running build_ext
Loading:
DEBUG: /home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/xla/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'llvm-raw' because it already exists.
DEBUG: /home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'com_google_absl' because it already exists.
DEBUG: /home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'pybind11_bazel' because it already exists.
DEBUG: /home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'com_google_protobuf' because it already exists.
DEBUG: /home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'com_google_googletest' because it already exists.
DEBUG: /home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'com_github_gflags_gflags' because it already exists.
DEBUG: /home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'com_github_grpc_grpc' because it already exists.
DEBUG: /home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'zlib' because it already exists.
DEBUG: /home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'nvtx_archive' because it already exists.
DEBUG: /home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'pybind11' because it already exists.
DEBUG: /home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'upb' because it already exists.
DEBUG: /home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/xla/third_party/repo.bzl:132:14:
Warning: skipping import of repository 'jsoncpp_git' because it already exists.
Loading:
Loading: 1 packages loaded
Analyzing: 3 targets (3 packages loaded, 0 targets configured)
INFO: Repository local_execution_config_python instantiated at:
/home/userA/project/sf/spu/WORKSPACE:72:15: in <toplevel>
/home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/xla/workspace2.bzl:123:19: in workspace
/home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/workspace2.bzl:617:19: in workspace
/home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/workspace2.bzl:67:27: in _tf_toolchains
/home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/tools/toolchains/remote_config/configs.bzl:6:28: in initialize_rbe_configs
/home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/tools/toolchains/remote_config/rbe_config.bzl:158:27: in _tensorflow_local_config
Repository rule local_python_configure defined at:
/home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/third_party/py/python_configure.bzl:276:41: in <toplevel>
ERROR: An error occurred during the fetch of repository 'local_execution_config_python':
Traceback (most recent call last):
File "/home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/third_party/py/python_configure.bzl", line 213, column 39, in _create_local_python_repository
numpy_include = _get_numpy_include(repository_ctx, python_bin) + "/numpy"
File "/home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/third_party/py/python_configure.bzl", line 187, column 19, in _get_numpy_include
return execute(
File "/home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/third_party/remote_config/common.bzl", line 230, column 13, in execute
fail(
Error in fail: Problem getting numpy include path.
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
Is numpy installed?
ERROR: /home/userA/project/sf/spu/WORKSPACE:72:15: fetching local_python_configure rule //external:local_execution_config_python: Traceback (most recent call last):
File "/home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/third_party/py/python_configure.bzl", line 213, column 39, in _create_local_python_repository
numpy_include = _get_numpy_include(repository_ctx, python_bin) + "/numpy"
File "/home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/third_party/py/python_configure.bzl", line 187, column 19, in _get_numpy_include
return execute(
File "/home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/third_party/remote_config/common.bzl", line 230, column 13, in execute
fail(
Error in fail: Problem getting numpy include path.
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
Is numpy installed?
Analyzing: 3 targets (43 packages loaded, 34 targets configured)
INFO: Repository go_sdk instantiated at:
/home/userA/project/sf/spu/WORKSPACE:80:15: in <toplevel>
/home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/xla/workspace0.bzl:52:19: in workspace
/home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/tsl/workspace0.bzl:130:20: in workspace
/home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/com_github_grpc_grpc/bazel/grpc_extra_deps.bzl:56:27: in grpc_extra_deps
/home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/io_bazel_rules_go/go/private/sdk.bzl:431:28: in go_register_toolchains
/home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/io_bazel_rules_go/go/private/sdk.bzl:130:21: in go_download_sdk
Repository rule _go_download_sdk defined at:
/home/userA/.cache/bazel/_bazel_userA/faae206eac0fa7d05e2c25985585c89f/external/io_bazel_rules_go/go/private/sdk.bzl:117:35: in <toplevel>
ERROR: Analysis of target '//spu:init' failed; build aborted: Problem getting numpy include path.
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
Is numpy installed?
INFO: Elapsed time: 2.288s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (43 packages loaded, 34 targets configured)
Traceback (most recent call last):
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 138, in run
self._create_wheel_file(bdist_wheel)
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 341, in _create_wheel_file
files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 264, in _run_build_commands
self._run_build_subcommands()
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 291, in _run_build_subcommands
self.run_command(name)
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
self.distribution.run_command(command)
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 950, in run_command
super().run_command(command)
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "<string>", line 237, in run
File "<string>", line 208, in pip_run
File "<string>", line 164, in build
File "<string>", line 123, in bazel_invoke
File "/home/userA/miniconda3/envs/sf/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['bazel', 'build', '--verbose_failures', '-c', 'opt', '--config=avx', '--', '//spu:init', '//spu/utils:distributed', '//spu:api']' returned non-zero exit status 1.
/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py:972: _DebuggingTips: Problem in editable installation.
!!
********************************************************************************
An error happened while installing `spu` in editable mode.
The following steps are recommended to help debug this problem:
- Try to install the project normally, without using the editable mode.
Does the error still persist?
(If it does, try fixing the problem before attempting the editable mode).
- If you are using binary extensions, make sure you have all OS-level
dependencies installed (e.g. compilers, toolchains, binary libraries, ...).
- Try the latest version of setuptools (maybe the error was already fixed).
- If you (or your project dependencies) are using any setuptools extension
or customization, make sure they support the editable mode.
After following the steps above, if the problem still persists and
you think this is related to how setuptools handles editable installations,
please submit a reproducible example
(see https://stackoverflow.com/help/minimal-reproducible-example) to:
https://github.com/pypa/setuptools/issues
See https://setuptools.pypa.io/en/latest/userguide/development_mode.html for details.
********************************************************************************
!!
cmd_obj.run()
Build with extra flags = ['--verbose_failures', '-c', 'opt', '--config=avx']
Traceback (most recent call last):
File "/home/userA/miniconda3/envs/sf/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/home/userA/miniconda3/envs/sf/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/userA/miniconda3/envs/sf/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 273, in build_editable
return hook(wheel_directory, config_settings, metadata_directory)
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 459, in build_editable
return self._build_with_temp_dir(
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 403, in _build_with_temp_dir
self.run_setup()
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 503, in run_setup
super().run_setup(setup_script=setup_script)
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 318, in run_setup
exec(code, locals())
File "<string>", line 266, in <module>
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 117, in setup
return distutils.core.setup(**attrs)
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 184, in setup
return run_commands(dist)
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
dist.run_commands()
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 950, in run_command
super().run_command(command)
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 138, in run
self._create_wheel_file(bdist_wheel)
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 341, in _create_wheel_file
files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 264, in _run_build_commands
self._run_build_subcommands()
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/command/editable_wheel.py", line 291, in _run_build_subcommands
self.run_command(name)
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
self.distribution.run_command(command)
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 950, in run_command
super().run_command(command)
File "/tmp/pip-build-env-75c0zuwj/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "<string>", line 237, in run
File "<string>", line 208, in pip_run
File "<string>", line 164, in build
File "<string>", line 123, in bazel_invoke
File "/home/userA/miniconda3/envs/sf/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['bazel', 'build', '--verbose_failures', '-c', 'opt', '--config=avx', '--', '//spu:init', '//spu/utils:distributed', '//spu:api']' returned non-zero exit status 1.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building editable for spu
Failed to build spu
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (spu)Hi @grueyg
你是想 build 一个 wheel 包吗？直接用这个 script 吧
@anakinxc，谢谢，使用这个脚本可以正确build出whl包，也可以进行正确的安装。
但我想的是，将spu安装到当前目录(pip install -e .)，而不是python的site-packages里(pip install *.whl)。这样每次修改源代码后，只需要在当前目录下重新编译即可。
这样操作是可行的吗？还是说这样做，实际上没有每次修改后build成whl包再重新安装方便？
最后，直接pip install . 或 pip install -e .为什么不能正确执行呢？反而是先build成whl，再pip install就可以正确执行
anakinxc
commented
2 months ago anakinxc
commented
2 months ago 最后，直接
pip install .或pip install -e .为什么不能正确执行呢？反而是先build成whl，再pip install就可以正确执行
这个感觉是 pip env 的 setup 的问题
grueyg
commented
2 months ago 谢谢你的解答，我没有任何问题了
Issue Type
Build/Install
Modules Involved
Others
Have you reproduced the bug with SPU HEAD?
Yes
Have you searched existing issues?
Yes
SPU Version
6564c08
OS Platform and Distribution
Linux Ubuntu 18.04
Python Version
3.10
Compiler Version
GCC 11.4
Current Behavior?
从源码中编译spu的时候，出现大量错误。
我的gcc和g++的安装方式是
Standalone code to reproduce the issue
Relevant log output