search

microsoft / nni

An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.
https://nni.readthedocs.io
MIT License
14.07k stars 1.82k forks source link

Build from source on MacOS M1 failed #5142

Closed DDDOH closed 1 year ago

DDDOH commented 2 years ago

Describe the issue: I followed instructions from https://nni.readthedocs.io/zh/stable/notes/build_from_source.html, run command

conda create --name my_env
conda activate my_env
git clone https://github.com/microsoft/nni.git
cd nni
pip install --upgrade setuptools pip wheel
pip install jupyterlab==3.0.9
export NNI_RELEASE=2.0
python setup.py build_ts

I got error when running 'python setup.py build_ts', as shown below

running build_ts
# Downloading node.js from https://nodejs.org/dist/v16.14.2/node-v16.14.2-darwin-arm64.tar.xz 
# Extracting node.js 
# Downloading yarn from https://github.com/yarnpkg/yarn/releases/download/v1.22.10/yarn-v1.22.10.tar.gz 
# Extracting yarn 
# Building NNI manager 
# yarn  (path: ts/nni_manager) 
yarn install v1.22.10
[1/5] 🔍  Validating package.json...
[2/5] 🔍  Resolving packages...
[3/5] 🚚  Fetching packages...
[4/5] 🔗  Linking dependencies...
warning " > express-joi-validator@2.0.1" has incorrect peer dependency "joi@6.x.x".
warning " > @typescript-eslint/eslint-plugin@2.34.0" has incorrect peer dependency "@typescript-eslint/parser@^2.0.0".
warning " > @typescript-eslint/eslint-plugin@2.34.0" has incorrect peer dependency "eslint@^5.0.0 || ^6.0.0".
warning Workspaces can only be enabled in private projects.
[5/5] 🔨  Building fresh packages...
[1/3] ⢀ sqlite3
[2/3] ⢀ cpu-features
warning Error running install script for optional dependency: "/Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features: Command failed.
Exit code: 1
Command: node-gyp rebuild
Arguments: 
Directory: /Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features
Output:
gyp info it worked if it ends with ok
gyp info using node-gyp@9.1.0
gyp info using node@16.14.2 | darwin | arm64
gyp info find Python using Python version 3.10.4 found at \"/Users/shuffleofficial/opt/anaconda3/envs/ds_wgan/bin/python3\"
gyp info spawn /Users/shuffleofficial/opt/anaconda3/envs/ds_wgan/bin/python3
gyp info spawn args [
gyp info spawn args   '/Users/shuffleofficial/.config/yarn/global/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/shuffleofficial/.config/yarn/global/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/shuffleofficial/Library/Caches/node-gyp/16.14.2/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/Users/shuffleofficial/Library/Caches/node-gyp/16.14.2',
gyp info spawn args   '-Dnode_gyp_dir=/Users/shuffleofficial/.config/yarn/global/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/Users/shuffleofficial/Library/Caches/node-gyp/16.14.2/<(target_arch)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.'
gyp info spawn args ]
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
  ACTION Configuring dependencies /Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features/deps/cpu_features/build/Makefile
-- The C compiler identification is AppleClang 14.0.0.14000029
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Looking for dlfcn.h
-- Looking for dlfcn.h - found
-- Looking for getauxval
-- Looking for getauxval - not found
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features/deps/cpu_features/build
  TOUCH Release/obj.target/config_deps.stamp
  ACTION Building dependencies /Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features/deps/cpu_features/build/libcpu_features.a
[ 11%] Building C object CMakeFiles/utils.dir/src/filesystem.c.o
[ 22%] Building C object CMakeFiles/utils.dir/src/stack_line_reader.c.o
[ 33%] Building C object CMakeFiles/utils.dir/src/string_view.c.o
[ 33%] Built target utils
[ 44%] Building C object CMakeFiles/unix_based_hardware_detection.dir/src/hwcaps.c.o
[ 55%] Building C object CMakeFiles/unix_based_hardware_detection.dir/src/unix_features_aggregator.c.o
[ 55%] Built target unix_based_hardware_detection
[ 66%] Building C object CMakeFiles/cpu_features.dir/src/cpuinfo_arm.c.o
In file included from /Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features/deps/cpu_features/src/cpuinfo_arm.c:15:
/Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features/deps/cpu_features/include/cpuinfo_arm.h:118:2: error: \"Including cpuinfo_arm.h from a non-arm target.\"
#error \"Including cpuinfo_arm.h from a non-arm target.\"
 ^
1 error generated.
make[3]: *** [CMakeFiles/cpu_features.dir/src/cpuinfo_arm.c.o] Error 1
make[2]: *** [CMakeFiles/cpu_features.dir/all] Error 2
make[1]: *** [all] Error 2
make: *** [/Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features/deps/cpu_features/build/libcpu_features.a] Error 2
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/Users/shuffleofficial/.config/yarn/global/node_modules/node-gyp/lib/build.js:201:23)
gyp ERR! stack     at ChildProcess.emit (node:events:526:28)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:291:12)
gyp ERR! System Darwin 21.6.0
gyp ERR! command \"/Users/shuffleofficial/nni/nni_node/node\" \"/Users/shuffleofficial/.yarn/bin/node-gyp\" \"rebuild\"
gyp ERR! cwd /Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features
gyp ERR! node -v v16.14.2
gyp ERR! node-gyp -v v9.1.0
✨  Done in 39.75s.
# yarn build (path: ts/nni_manager) 
yarn run v1.22.10
$ tsc
/bin/sh: tsc: command not found
error Command failed with exit code 127.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Traceback (most recent call last):
  File "/Users/shuffleofficial/nni/setup.py", line 300, in <module>
    _setup()
  File "/Users/shuffleofficial/nni/setup.py", line 93, in _setup
    setuptools.setup(
  File "/Users/shuffleofficial/opt/anaconda3/envs/ds_wgan/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "/Users/shuffleofficial/opt/anaconda3/envs/ds_wgan/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/Users/shuffleofficial/opt/anaconda3/envs/ds_wgan/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/Users/shuffleofficial/opt/anaconda3/envs/ds_wgan/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
    self.run_command(cmd)
  File "/Users/shuffleofficial/opt/anaconda3/envs/ds_wgan/lib/python3.10/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "/Users/shuffleofficial/opt/anaconda3/envs/ds_wgan/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    cmd_obj.run()
  File "/Users/shuffleofficial/nni/setup.py", line 224, in run
    setup_ts.build(release)
  File "/Users/shuffleofficial/nni/setup_ts.py", line 53, in build
    compile_ts(release)
  File "/Users/shuffleofficial/nni/setup_ts.py", line 173, in compile_ts
    _yarn('ts/nni_manager', 'build')
  File "/Users/shuffleofficial/nni/setup_ts.py", line 272, in _yarn
    subprocess.run([str(_yarn_path), *args], cwd=path, check=True, env=_yarn_env)
  File "/Users/shuffleofficial/opt/anaconda3/envs/ds_wgan/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/shuffleofficial/nni/toolchain/yarn/bin/yarn', 'build']' returned non-zero exit status 127.

Environment:

How to reproduce it?: On a macos with M1 chip, run the terminal command above.

Lijiaoa commented 2 years ago

hi @DDDOH Could you use nni v2.9?

DDDOH commented 2 years ago

hi @DDDOH Could you use nni v2.9?

With export NNI_RELEASE=2.9?

cruiseliu commented 2 years ago

Please try following command in ts/nni_manager folder:

/Users/shuffleofficial/nni/toolchain/yarn/bin/yarn tsc

It should find tsc in ts/nni_manager/node_modules/.bin/tsc, which is a symlink to ts/nni_manager/node_modules/typescript/bin/tsc. And the later file should be executable (should show rwx in ls -l result).

If tsc is not there, please remove node_modules folder and run /Users/shuffleofficial/nni/toolchain/yarn/bin/yarn to redownload.

If the file is there but cannot be found, then there might be something wrong with yarn. In this case, upgrade yarn_version in setup_ts.py to 1.22.19 and try again. (Remember to clean first)

cruiseliu commented 2 years ago

To further debug it, you can add a line "debug": "echo $PATH", to package.json line 6 ("scripts" section), and then run yarn debug to check search paths. It should include /Users/shuffleofficial/nni/ts/nni_manager/node_modules/.bin

DDDOH commented 2 years ago

https://github.com/microsoft/nni/issues/5142#issuecomment-1260335580 I upgrade yarn_version in setup_ts.py to 1.22.19, and run python setup.py clean, then python setup.py build_ts (and failed again with error message exactly the same as before).

Please try following command in ts/nni_manager folder: /Users/shuffleofficial/nni/toolchain/yarn/bin/yarn tsc

gets 

yarn run v1.22.19
error Couldn't find a package.json file in "/Users/shuffleofficial/nni"
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

And there is no tsc in the ts/nni_manager/node_modules/.bin/ folder:

image

I then run /Users/shuffleofficial/nni/toolchain/yarn/bin/yarn, there's still not a tsc in the ts/nni_manager/node_modules/.bin/ folder.

I'm not familiar with node and javascript staff, so I actually don't quite figure out which package.json you are referring to. If more information is needed, please let me know.

But I finally found one way to run nni on a M1 macbook: create a x86 conda environment, then pip install nni, and unsurprisingly this method is really slow but is useful for debugging.

scarlett2018 commented 2 years ago

#5142 (comment) I upgrade yarn_version in setup_ts.py to 1.22.19, and run python setup.py clean, then python setup.py build_ts (and failed again with error message exactly the same as before).

Please try following command in ts/nni_manager folder: /Users/shuffleofficial/nni/toolchain/yarn/bin/yarn tsc

gets

yarn run v1.22.19
error Couldn't find a package.json file in "/Users/shuffleofficial/nni"
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

And there is no tsc in the ts/nni_manager/node_modules/.bin/ folder: image

I then run /Users/shuffleofficial/nni/toolchain/yarn/bin/yarn, there's still not a tsc in the ts/nni_manager/node_modules/.bin/ folder.

I'm not familiar with node and javascript staff, so I actually don't quite figure out which package.json you are referring to. If more information is needed, please let me know.

But I finally found one way to run nni on a M1 macbook: create a x86 conda environment, then pip install nni, and unsurprisingly this method is really slow but is useful for debugging.

@cruiseliu - more suggestions?

liuzhe-lz commented 2 years ago

Please run yarn in ts/nni_manager directory (not nni directory), and then check:

  1. Whether there is a typescript directory in node_modules.
  2. Whether there is typescript@^4.3.2 (or other version) entry yarn.lock file.
  3. Whether there is "typescript": "^4.3.2" in package.json file.

If the first answer is yes, please try manually make the symlink and run yarn build to compile. If the first answer is no and the second is yes, then it's a yarnpkg bug and we should report it to their issue tracker. Otherwise, please git checkout master and try again.

Lijiaoa commented 2 years ago

hi @DDDOH Did you still have the install issue? You could follow the @cruiseliu suggestions.