Open weiji14 opened 2 years ago
NNI can't be installed in this way because nni_node
needs to be pre-built. See build from source.
Thanks, I managed to do git clone https://github.com/microsoft/nni.git
and then pip install --verbose --editable nni/
, and it seems to work. That said, I noticed lots of warnings:
Installing collected packages: nni
Running setup.py develop for nni
Running command python setup.py develop
running develop
/home/username/miniconda3/envs/projectname/lib/python3.9/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
/home/username/miniconda3/envs/projectname/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
# Building NNI manager
yarn install v1.22.10
[1/5] Validating package.json...
[2/5] Resolving packages...
success Already up-to-date.
Done in 0.19s.
yarn run v1.22.10
$ tsc
Done in 3.05s.
# Building web UI
yarn install v1.22.10
[1/4] Resolving packages...
[2/4] Fetching packages...
info fsevents@2.3.2: The platform "linux" is incompatible with this module.
info "fsevents@2.3.2" is an optional dependency and failed compatibility check. Excluding it from installation.
info fsevents@1.2.13: The platform "linux" is incompatible with this module.
info "fsevents@1.2.13" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
warning " > postcss-flexbugs-fixes@5.0.2" has unmet peer dependency "postcss@^8.1.4".
warning " > postcss-loader@6.2.1" has unmet peer dependency "postcss@^7.0.0 || ^8.0.1".
warning " > postcss-normalize@10.0.1" has unmet peer dependency "postcss@>= 8".
warning "postcss-normalize > postcss-browser-comments@4.0.0" has unmet peer dependency "postcss@>=8".
warning " > webpack-manifest-plugin@3.2.0" has incorrect peer dependency "webpack@^4.44.2".
warning " > eslint-config-react-app@6.0.0" has incorrect peer dependency "@typescript-eslint/eslint-plugin@^4.0.0".
warning " > eslint-config-react-app@6.0.0" has incorrect peer dependency "@typescript-eslint/parser@^4.0.0".
warning " > eslint-config-react-app@6.0.0" has unmet peer dependency "babel-eslint@^10.0.0".
warning Workspaces can only be enabled in private projectnames.
[4/4] Building fresh packages...
success Saved lockfile.
Done in 2.79s.
yarn run v1.22.10
$ node --max-old-space-size=3072 scripts/developmentBuild.js
Creating an development build...
Compiled successfully.
Done in 5.86s.
# Building JupyterLab extension
yarn install v1.22.10
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.14s.
yarn run v1.22.10
$ tsc && jupyter labextension build .
Building extension in .
An error occurred.
ValueError: Extensions require a devDependency on @jupyterlab/builder@^3.4.0, you have a dependency on 3.0.9
See the log file for details: /tmp/jupyterlab-debug-1nf3i9e3.log
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
# Failed to build JupyterLab extension, skip for develop mode
# Traceback (most recent call last):
File "/home/username/projectname/nni/setup_ts.py", line 188, in compile_ts
_yarn('ts/jupyter_extension', 'build')
File "/home/username/projectname/nni/setup_ts.py", line 267, in _yarn
subprocess.run([str(_yarn_path), *args], cwd=path, check=True, env=_yarn_env)
File "/home/username/miniconda3/envs/projectname/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/username/projectname/nni/toolchain/yarn/bin/yarn', 'build']' returned non-zero exit status 1.
# Creating symlinks
running egg_info
writing nni.egg-info/PKG-INFO
writing dependency_links to nni.egg-info/dependency_links.txt
writing entry points to nni.egg-info/entry_points.txt
writing requirements to nni.egg-info/requires.txt
writing top-level names to nni.egg-info/top_level.txt
package init file 'nni/algorithms/compression/tensorflow/__init__.py' not found (or not a regular file)
package init file 'nni/algorithms/compression/v2/__init__.py' not found (or not a regular file)
package init file 'nni/compression/pytorch/quantization/__init__.py' not found (or not a regular file)
reading manifest file 'nni.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'nni.egg-info/SOURCES.txt'
running build_ext
Creating /home/username/miniconda3/envs/projectname/lib/python3.9/site-packages/nni.egg-link (link to .)
Adding nni 999.dev0 to easy-install.pth file
Installing nnictl script to /home/username/miniconda3/envs/projectname/bin
Installed /home/username/projectname/nni
Successfully installed nni-999.dev0
In particular, setup.py
based installs will be deprecated in the future. Is there a timeline for nni
to move to using build
and/or other standards-based tools that is compliant with PEP517? I can help open a PR for this, though I'm not familiar with the node
/javascript stuff, but I can give it a shot.
@liuzhe-lz Any comments on this?
P.S. When I looked at pyright the other day, I noticed an interesting way, that is to publish the ts package on npm, and maintain the python wrapper in another repo. We can also try that. That might solve the problem that we need a customized python setup.py develop
.
I don't like setuptools either but I'm afraid Python team has not worked out a serious alternative solution yet.
PEP 517 requires backends to create .whl
files, but hey, where is the API? For us 3rd-party users wheel
package is for command line use only. AFAIK the only documented programmatic way to build wheel package is via setuptools.setup()
. Correct me if I'm wrong. I'll be excited for that.
The Python packaging tools have actually matured a bit more recently. A project I help maintain switched 2 months ago (https://github.com/GenericMappingTools/pygmt/issues/1797) from using setuptools to PyPA's build
to build source distributions (sdist) and binary distributions (wheels), and we haven't had any issues so far. Specifically, we switched the build command from python setup.py sdist
to python -m build
. See also https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html#summary
Now I'm not sure what the situation is with building a typescript package, but as @matluster mentioned above, maybe it's possible to separate the Python and Typescript components of nni
? How closely integrated is the ts package here?
The Python packaging tools have actually matured a bit more recently. A project I help maintain switched 2 months ago (GenericMappingTools/pygmt#1797) from using setuptools to PyPA's
build
to build source distributions (sdist) and binary distributions (wheels), and we haven't had any issues so far. Specifically, we switched the build command frompython setup.py sdist
topython -m build
. See also https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html#summaryNow I'm not sure what the situation is with building a typescript package, but as @matluster mentioned above, maybe it's possible to separate the Python and Typescript components of
nni
? How closely integrated is the ts package here?
To my understanding build
is a frontend, but we are missing the backend.
NNI is not a "static" repo that you copy source files into an archive and everything done. Instead we have Python code as build hooks. Unfortunately seems the only Python build tool that supports customizing with Python code is setuptools.
Pyright is another story. It is a nodejs project with Python files. It is easy to organize a hybrid project in that way, since the nodejs toolchain has absolutely mature support for build hooks. The problem is contributors will then need to develop the project in nodejs way. They have to setup nodejs runtime on their own and install the project with npx dev
. The Python devs will not be happy about that.
Well there is another approach.
Tough not well documented, .whl
format is relatively straightforward and it's not that hard to implement a non-full-featured backend by ourselves. The NNI repo should not contain this kind of code, but maybe we can host the tool on our devs' personal repo.
To my understanding
build
is a frontend, but we are missing the backend.
Came across this backend
library called Hatch (https://hatch.pypa.io/1.5). The Jupyter folks seem to have some success using it recently to build their Javascript assets (see https://blog.jupyter.org/packaging-for-jupyter-in-2022-c7be64c38926?gi=dcb04ade667a and https://github.com/jupyterlab/hatch-jupyter-builder). Haven't tried it personally, but just thought it might be of interest.
Describe the issue:
Hi there, just starting to use
nni
and was interested in trying some features in the master branch, but couldn't get the install to work. The error message iserror: package directory 'nni_node' does not exist
.Environment:
Configuration:
Log message:
How to reproduce it?:
Run
pip install git+https://github.com/microsoft/nni.git
. This produces the error:For reference, I'm using
pip=22.0.4
,setuptools=62.1.0
,wheel=0.37.1
.