IBM / powerai

This repo contains ancillary information used to assist users of IBM Watson Machine Learning Community Edition. This repo will contain How To's, Readme's, Dockerfiles, etc. that can be consumed by users looking to get started.
BSD 2-Clause "Simplified" License
56 stars 54 forks source link

Can't install python packages with either pip or conda during Docker build #277

Open jennyfolkesson opened 2 years ago

jennyfolkesson commented 2 years ago

Hi, I've been trying to build a Docker image on an IBM power9 server using ibmcom/powerai:1.6.0-all-ubuntu18.04-py3 as a base. I've tried adding python packages using both pip and conda but neither one works, and I'm at a loss. It's hard to find answers by googling the errors since not many use the ppc64le architecture, so any help you can give me would be greatly appreciated. Thank you.

Below is my Dockerfile for installing packages with conda, which during build hangs with the message: "Solving environment: ...working... "

`FROM ibmcom/powerai:1.6.0-all-ubuntu18.04-py3

ENV USER_NAME pwrai ENV WORK_DIR /home/${USER_NAME}/microDL

RUN ["/bin/bash", "-c", "cd /opt/anaconda3/bin && source activate base && IBM_POWERAI_LICENSE_ACCEPT=yes ./accept-powerai-license.sh" ]

RUN sudo /bin/ln -sf /bin/bash /bin/sh

RUN sudo apt-get update && sudo apt-get install -yq --no-install-recommends \ software-properties-common && \ add-apt-repository universe RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections \ && sudo apt-get update && sudo apt-get install -yq --no-install-recommends \ build-essential \ apt-utils \ libcupti-dev \ ca-certificates \ cmake \ libfreetype6-dev \ wget \ git \ pkg-config \ tmux \ graphviz \ vim

RUN sudo apt-get clean && \ sudo apt-get autoremove && \ sudo rm -rf /var/lib/apt/lists/*

ENV PATH /opt/anaconda3/bin:${PATH} RUN sudo ln -s /opt/anaconda3/etc/profile.d/conda.sh /etc/profile.d/conda.sh ADD conda_environment_docker.yml /tmp/conda.yml RUN conda update -n base -c defaults --override-channels conda && \ conda env create --file=/tmp/conda.yml && \ conda install tensorflow-gpu==1.13.1 && \ conda activate micro_dl

WORKDIR ${WORK_DIR} ENV PYTHONPATH ${WORK_DIR} RUN sudo chown ${USER_NAME} ${WORK_DIR} RUN sudo chgrp ${USER_NAME} ${WORK_DIR}

EXPOSE 8888 6006`

jennyfolkesson commented 2 years ago

Here is my Dockerfile for pip installation of packages:

`FROM ibmcom/powerai:1.6.0-all-ubuntu18.04-py3

ENV USER_NAME pwrai ENV WORK_DIR /home/${USER_NAME}/microDL USER root ENV PATH /root/.local/bin:${PATH}

RUN ["/bin/bash", "-c", "cd /opt/anaconda3/bin && source activate base && IBM_POWERAI_LICENSE_ACCEPT=yes ./accept-powerai-license.sh" ]

RUN sudo /bin/ln -sf /bin/bash /bin/sh

RUN apt-get update && apt-get install -y \ software-properties-common && \ add-apt-repository universe RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections \ && sudo apt-get update && sudo apt-get install -yq --no-install-recommends \ build-essential \ apt-utils \ libcupti-dev \ ca-certificates \ cmake \ libfreetype6-dev \ libpng-dev \ wget \ git \ pkg-config \ tmux \ graphviz \ vim \

python3-pip \

     python3-dev \
     python3-distutils \
     python3-venv

RUN sudo apt-get clean && \ sudo apt-get autoremove && \ sudo rm -rf /var/lib/apt/lists/*

RUN echo $PATH RUN wget https://bootstrap.pypa.io/pip/3.6/get-pip.py && \ python3.6 get-pip.py --target $(python3 -c 'import sys; print(sys.path[-1])') && \ rm get-pip.py && \

python3.6 -m venv env && source ./env/bin/activate && \ sudo python3.6 -m pip install --user --upgrade --force pip setuptools wheel && \ python3.6 -m pip install --user --upgrade Cython==0.29.10 && \ python3.6 -m pip install --user --upgrade numpy && \ python3.6 -m pip install --user --upgrade scipy && \ python3.6 -m pip install --user --upgrade scikit-image && \ python3.6 -m pip install --user --upgrade scikit-learn && \ python3.6 -m pip install --user --upgrade opencv-python && \ python3.6 -m pip install --user --upgrade pandas && \ python3.6 -m pip install --user --upgrade natsort nose protobuf pydot PyYAML testfixtures tqdm zarr && \

python3.6 -m pip install --user --upgrade --no-cache-dir keras==2.1.6 && \ python3.6 -m pip install --user --upgrade --no-cache-dir jupyter && \ python3.6 -m pip install --user --upgrade matplotlib

WORKDIR ${WORK_DIR} ENV PYTHONPATH ${WORK_DIR} RUN sudo chown ${USER_NAME} ${WORK_DIR} RUN sudo chgrp ${USER_NAME} ${WORK_DIR}

EXPOSE 8888 6006`

jennyfolkesson commented 2 years ago

And below is the error message I get trying to install packages with pip:

`Collecting pip<22.0 Downloading pip-21.3.1-py3-none-any.whl (1.7 MB) Collecting setuptools Downloading setuptools-59.6.0-py3-none-any.whl (952 kB) Collecting wheel Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB) Installing collected packages: wheel, setuptools, pip Successfully installed pip-21.3.1 setuptools-59.6.0 wheel-0.37.1 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv Collecting pip Using cached pip-21.3.1-py3-none-any.whl (1.7 MB) Collecting setuptools Using cached setuptools-59.6.0-py3-none-any.whl (952 kB) Collecting wheel Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB) Installing collected packages: wheel, setuptools, pip WARNING: The script wheel is installed in '/root/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The scripts pip, pip3 and pip3.6 are installed in '/root/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv Successfully installed pip-21.3.1 setuptools-59.6.0 wheel-0.37.1 Collecting Cython==0.29.10 Downloading https://files.pythonhosted.org/packages/e7/2e/aee8dfff93c7f5d20461ddcefd1da3d43bab18e1666a7777f4d9dbe94065/Cython-0.29.10.tar.gz (2.0MB) Building wheels for collected packages: Cython Running setup.py bdist_wheel for Cython: started Failed building wheel for Cython Running setup.py bdist_wheel for Cython: finished with status 'error' Complete output from command /env/bin/python3.6 -u -c "import setuptools, tokenize;file='/tmp/pip-build-x02aauu3/Cython/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d /tmp/tmp420xt618pip-wheel- --python-tag cp36: Unable to find pgen, not compiling formal grammar. usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: -c --help [cmd1 cmd2 ...] or: -c --help-commands or: -c cmd --help

error: invalid command 'bdist_wheel'


Running setup.py clean for Cython Failed to build Cython Installing collected packages: Cython Running setup.py install for Cython: started Running setup.py install for Cython: still running... Running setup.py install for Cython: finished with status 'done' Successfully installed Cython-0.29.10 Collecting numpy Downloading https://files.pythonhosted.org/packages/51/60/3f0fe5b7675a461d96b9d6729beecd3532565743278a9c3fe6dd09697fa7/numpy-1.19.5.zip (7.3MB) Building wheels for collected packages: numpy Running setup.py bdist_wheel for numpy: started Failed building wheel for numpy Running setup.py bdist_wheel for numpy: finished with status 'error' Complete output from command /env/bin/python3.6 -u -c "import setuptools, tokenize;file='/tmp/pip-build-hgmtxb0b/numpy/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d /tmp/tmp6uxnrillpip-wheel- --python-tag cp36: Running from numpy source directory. Cythonizing sources Processing numpy/random/_bounded_integers.pxd.in Processing numpy/random/_mt19937.pyx Traceback (most recent call last): File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 59, in process_pyx from Cython.Compiler.Version import version as cython_version ModuleNotFoundError: No module named 'Cython'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 235, in main() File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 231, in main find_process_files(root_dir) File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 222, in find_process_files process(root_dir, fromfile, tofile, function, hash_db) File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 188, in process processor_function(fromfile, tofile) File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 64, in process_pyx raise OSError('Cython needs to be installed in Python as a module') OSError: Cython needs to be installed in Python as a module Traceback (most recent call last): File "", line 1, in File "/tmp/pip-build-hgmtxb0b/numpy/setup.py", line 508, in setup_package() File "/tmp/pip-build-hgmtxb0b/numpy/setup.py", line 488, in setup_package generate_cython() File "/tmp/pip-build-hgmtxb0b/numpy/setup.py", line 285, in generate_cython raise RuntimeError("Running cythonize failed!") RuntimeError: Running cythonize failed!


Running setup.py clean for numpy Failed cleaning build dir for numpy Complete output from command /env/bin/python3.6 -u -c "import setuptools, tokenize;file='/tmp/pip-build-hgmtxb0b/numpy/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" clean --all: Running from numpy source directory.

setup.py clean is not supported, use one of the following instead:

- `git clean -xdf` (cleans all files)
- `git clean -Xdf` (cleans all versioned files, doesn't touch
                    files that aren't checked into the git repo)

Add --force to your command to use it anyway if you must (unsupported).


Failed to build numpy Installing collected packages: numpy Running setup.py install for numpy: started Running setup.py install for numpy: finished with status 'error' Complete output from command /env/bin/python3.6 -u -c "import setuptools, tokenize;file='/tmp/pip-build-hgmtxb0b/numpy/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-wxeof2ob-record/install-record.txt --single-version-externally-managed --compile --install-headers /env/include/site/python3.6/numpy --user --prefix=: Running from numpy source directory.

Note: if you need reliable uninstall behavior, then install
with pip instead of using `setup.py install`:

  - `pip install .`       (from a git repo or downloaded source
                           release)
  - `pip install numpy`   (last NumPy release on PyPi)

Cythonizing sources
numpy/random/_bounded_integers.pxd.in has not changed
Processing numpy/random/_mt19937.pyx
Traceback (most recent call last):
  File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 59, in process_pyx
    from Cython.Compiler.Version import version as cython_version
ModuleNotFoundError: No module named 'Cython'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 235, in <module>
    main()
  File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 231, in main
    find_process_files(root_dir)
  File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 222, in find_process_files
    process(root_dir, fromfile, tofile, function, hash_db)
  File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 188, in process
    processor_function(fromfile, tofile)
  File "/tmp/pip-build-hgmtxb0b/numpy/tools/cythonize.py", line 64, in process_pyx
    raise OSError('Cython needs to be installed in Python as a module')
OSError: Cython needs to be installed in Python as a module
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-build-hgmtxb0b/numpy/setup.py", line 508, in <module>
    setup_package()
  File "/tmp/pip-build-hgmtxb0b/numpy/setup.py", line 488, in setup_package
    generate_cython()
  File "/tmp/pip-build-hgmtxb0b/numpy/setup.py", line 285, in generate_cython
    raise RuntimeError("Running cythonize failed!")
RuntimeError: Running cythonize failed!

----------------------------------------

Command "/env/bin/python3.6 -u -c "import setuptools, tokenize;file='/tmp/pip-build-hgmtxb0b/numpy/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-wxeof2ob-record/install-record.txt --single-version-externally-managed --compile --install-headers /env/include/site/python3.6/numpy --user --prefix=" failed with error code 1 in /tmp/pip-build-hgmtxb0b/numpy/`

alexlang74 commented 2 years ago

Hi @jennyfolkesson , to my knowledge, these PowerAI channels are no longer supported . We are building much more recent versions of data science libraries as part of https://github.com/open-ce/open-ce .

Our University Partners at Oregon State University provide conda channels with these packages. See https://osuosl.org/services/powerdev/opence/#current-release on a description how to install these packages

eric-aquaronne commented 2 years ago

Hi @jennyfolkesson , Alex is correct : you should use OpenCE recipes. If you need a pre built set, beside the OSU 'academic build', we have a reliable build for our client made by Rocket: see the RocketCE conda channel. I would be very interested to know your use case are (what libraries do you use....) we are working to make sure our Power CPUs features (e.g. inside CPU matrix accelerators) are used to the best : can you reach me at eric.aquaronne@fr.ibm.com ? Thank you

jennyfolkesson commented 2 years ago

Hi Eric, We were stuck with the old docker channels because we had a repo using an old version of TensorFlow that we were using on a power9 server. We're right now in the process of updating the repo to pytorch. After the update, I'll reach out to you if we have issues with using OpenCE. Thank you!