pop-os / tensorman

Utility for easy management of Tensorflow containers
GNU General Public License v3.0
206 stars 16 forks source link

Installing other python packages for use w/ container? #12

Closed drscotthawley closed 4 years ago

drscotthawley commented 4 years ago

Guys, I know that what you've done must be amazing, but it's new to me and I'm stuck-- I'm well acquainted with managing python environments via anaconda and virtualenv, but when I get into the docker container a bunch of things (man, sudo, git) don't seem to work. My current issue is that I'm trying to install and use the opencv python package, i.e.

pip3 install --user opencv-python

Which seems to go great: installs fine, and if I run it in my 'main environment' it's acknowledged by python, but whenever I try to run python within the docker image, I get 'no module named cv2'. If I try to install opencv within the docker image, the installation seems to go through, but then I end up getting the same error when I try to import it in python. I've tried installing it as root too, both inside and outside of the "tf-docker" environment/machine/whatever.

myusername@system76:~$ tensorman run --python3 --gpu bash
"docker" "run" "-u" "1000:1000" "--gpus=all" "-e" "HOME=/project" "-it" "--rm" "-v" "/home/shawley/SPNet:/project" "-w" "/project" "tensorflow/tensorflow:latest-gpu-py3" "bash"

________                               _______________                
___  __/__________________________________  ____/__  /________      __
__  /  _  _ \_  __ \_  ___/  __ \_  ___/_  /_   __  /_  __ \_ | /| / /
_  /   /  __/  / / /(__  )/ /_/ /  /   _  __/   _  / / /_/ /_ |/ |/ / 
/_/    \___//_/ /_//____/ \____//_/    /_/      /_/  \____/____/|__/

You are running this container as user with ID 1000 and group 1000,
which should map to the ID and group for your user on the Docker host. Great!

tf-docker ~ > python
Python 3.6.8 (default, Aug 20 2019, 17:12:48) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'cv2'
>>> quit
Use quit() or Ctrl-D (i.e. EOF) to exit
>>> quit()
tf-docker ~ > pip3 install --user opencv-python
Requirement already satisfied: opencv-python in ./.local/lib/python3.6/site-packages (4.1.2.30)
Requirement already satisfied: numpy>=1.11.3 in /usr/local/lib/python3.6/dist-packages (from opencv-python) (1.17.2)
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
tf-docker ~ > python3
Python 3.6.8 (default, Aug 20 2019, 17:12:48) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'cv2'

That's the error. Here's a little extra checking

tf-docker ~ > env | grep PYTHON
tf-docker ~ > env | grep LIB
LD_LIBRARY_PATH=/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64

tf-docker ~ > exit
exit
myusername@system76:~$ python3
Python 3.7.5 (default, Nov 20 2019, 09:21:52) 
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> 

^ No problem as long as it's not within tf-docker.

Can you help? Thanks. :-)

drscotthawley commented 4 years ago

Trying it one more time:

@system76:~$ tensorman run --python3 --gpu bash
"docker" "run" "-u" "1000:1000" "--gpus=all" "-e" "HOME=/project" "-it" "--rm" "-v" "/home/shawley:/project" "-w" "/project" "tensorflow/tensorflow:latest-gpu-py3" "bash"

________                               _______________                
___  __/__________________________________  ____/__  /________      __
__  /  _  _ \_  __ \_  ___/  __ \_  ___/_  /_   __  /_  __ \_ | /| / /
_  /   /  __/  / / /(__  )/ /_/ /  /   _  __/   _  / / /_/ /_ |/ |/ / 
/_/    \___//_/ /_//____/ \____//_/    /_/      /_/  \____/____/|__/

You are running this container as user with ID 1000 and group 1000,
which should map to the ID and group for your user on the Docker host. Great!

I have no name!@8663dfff4176:~$ which python
/usr/local/bin/python
I have no name!@8663dfff4176:~$ pip install opencv-python
Collecting opencv-python
  Downloading https://files.pythonhosted.org/packages/c0/a9/9828dfaf93f40e190ebfb292141df6b7ea1a2d57b46263e757f52be8589f/opencv_python-4.1.2.30-cp36-cp36m-manylinux1_x86_64.whl (28.3MB)
     |████████████████████████████████| 28.3MB 40.1MB/s 
Requirement already satisfied: numpy>=1.11.3 in /usr/local/lib/python3.6/dist-packages (from opencv-python) (1.17.2)
Installing collected packages: opencv-python
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/local/lib/python3.6/dist-packages/cv2'
Consider using the `--user` option or check the permissions.

WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

I have no name!@8663dfff4176:~$ pip3 install --user opencv-python
Collecting opencv-python
  Using cached https://files.pythonhosted.org/packages/c0/a9/9828dfaf93f40e190ebfb292141df6b7ea1a2d57b46263e757f52be8589f/opencv_python-4.1.2.30-cp36-cp36m-manylinux1_x86_64.whl
Requirement already satisfied: numpy>=1.11.3 in /usr/local/lib/python3.6/dist-packages (from opencv-python) (1.17.2)
Installing collected packages: opencv-python
Successfully installed opencv-python-4.1.2.30
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
I have no name!@8663dfff4176:~$ python3
Python 3.6.8 (default, Aug 20 2019, 17:12:48) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/project/.local/lib/python3.6/site-packages/cv2/__init__.py", line 3, in <module>
    from .cv2 import *
ImportError: libSM.so.6: cannot open shared object file: No such file or directory

Ok...that's different, but still stuck.

mmstick commented 4 years ago

See https://github.com/pop-os/tensorman#creating-a-custom-image on how to create a custom image.

As for Python packages, any packages accessible from the working directory are accessible in the container.

drscotthawley commented 4 years ago

Thanks for your help Michael! I didn't realize "creating a custom image" was the name for what I was trying to do! (That sounded more 'advanced' for some reason.)

So now I've implemented those instructions: installed the packages I wanted to a named container, saved that container to an image, then ran the image in a new instance of tensorman, and... still there's no cv2.

In one shell:

$ tensorman run --gpu --python3 --root --name SPNET bash
"docker" "run" "-u" "root" "--name" "tensorman-SPNET" "--gpus=all" "-e" "HOME=/project" "-it" "--rm" "-v" "/home/shawley/SPNet:/project" "-w" "/project" "tensorflow/tensorflow:latest-gpu-py3" "bash"

________                               _______________                
___  __/__________________________________  ____/__  /________      __
__  /  _  _ \_  __ \_  ___/  __ \_  ___/_  /_   __  /_  __ \_ | /| / /
_  /   /  __/  / / /(__  )/ /_/ /  /   _  __/   _  / / /_/ /_ |/ |/ / 
/_/    \___//_/ /_//____/ \____//_/    /_/      /_/  \____/____/|__/

WARNING: You are running this container as root, which can cause new files in
mounted volumes to be created as the root user on your host machine.

To avoid this, run the container by specifying your user's userid:

$ docker run -u $(id -u):$(id -g) args...

tf-docker ~ > pip3 install opencv-python
WARNING: The directory '/project/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
WARNING: The directory '/project/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied: opencv-python in ./.local/lib/python3.6/site-packages (4.1.2.30)
Requirement already satisfied: numpy>=1.11.3 in /usr/local/lib/python3.6/dist-packages (from opencv-python) (1.17.2)
tf-docker ~ > pip3 install keras
WARNING: The directory '/project/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
WARNING: The directory '/project/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied: keras in ./.local/lib/python3.6/site-packages (2.3.1)
Requirement already satisfied: six>=1.9.0 in /usr/lib/python3/dist-packages (from keras) (1.11.0)
Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from keras) (1.0.8)
Requirement already satisfied: pyyaml in ./.local/lib/python3.6/site-packages (from keras) (5.2)
Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras) (2.10.0)
Requirement already satisfied: numpy>=1.9.1 in /usr/local/lib/python3.6/dist-packages (from keras) (1.17.2)
Requirement already satisfied: scipy>=0.14 in ./.local/lib/python3.6/site-packages (from keras) (1.4.1)
Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from keras) (1.1.0)
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
tf-docker ~ > pip install Pillow
WARNING: The directory '/project/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
WARNING: The directory '/project/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting Pillow
  Downloading https://files.pythonhosted.org/packages/10/5c/0e94e689de2476c4c5e644a3bd223a1c1b9e2bdb7c510191750be74fa786/Pillow-6.2.1-cp36-cp36m-manylinux1_x86_64.whl (2.1MB)
     |████████████████████████████████| 2.1MB 1.5MB/s 
Installing collected packages: Pillow
Successfully installed Pillow-6.2.1
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
tf-docker ~ > pip install imageio colorama
WARNING: The directory '/project/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
WARNING: The directory '/project/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting imageio
  Downloading https://files.pythonhosted.org/packages/1a/de/f7f985018f462ceeffada7f6e609919fbcc934acd9301929cba14bc2c24a/imageio-2.6.1-py3-none-any.whl (3.3MB)
     |████████████████████████████████| 3.3MB 1.1MB/s 
Collecting colorama
  Downloading https://files.pythonhosted.org/packages/c9/dc/45cdef1b4d119eb96316b3117e6d5708a08029992b2fee2c143c7a0a5cc5/colorama-0.4.3-py2.py3-none-any.whl
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from imageio) (1.17.2)
Requirement already satisfied: pillow in /usr/local/lib/python3.6/dist-packages (from imageio) (6.2.1)
Installing collected packages: imageio, colorama
Successfully installed colorama-0.4.3 imageio-2.6.1
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
tf-docker ~ > pip install pandas numba
WARNING: The directory '/project/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
WARNING: The directory '/project/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting pandas
  Downloading https://files.pythonhosted.org/packages/52/3f/f6a428599e0d4497e1595030965b5ba455fd8ade6e977e3c819973c4b41d/pandas-0.25.3-cp36-cp36m-manylinux1_x86_64.whl (10.4MB)
     |████████████████████████████████| 10.4MB 38kB/s 
Collecting numba
  Downloading https://files.pythonhosted.org/packages/53/34/22b6c2ded558976b5367be01b851ae679a0d1ba994de002d54afe84187b5/numba-0.46.0-cp36-cp36m-manylinux1_x86_64.whl (3.6MB)
     |████████████████████████████████| 3.6MB 18.1MB/s 
Collecting pytz>=2017.2 (from pandas)
  Downloading https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl (509kB)
     |████████████████████████████████| 512kB 14.7MB/s 
Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from pandas) (1.17.2)
Collecting python-dateutil>=2.6.1 (from pandas)
  Downloading https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl (227kB)
     |████████████████████████████████| 235kB 23.0MB/s 
Collecting llvmlite>=0.30.0dev0 (from numba)
  Downloading https://files.pythonhosted.org/packages/b3/93/924788871a889ead1c115ba2d43e67932a8865d38fdb179505c99e4bb575/llvmlite-0.30.0-cp36-cp36m-manylinux1_x86_64.whl (20.2MB)
     |████████████████████████████████| 20.2MB 24.5MB/s 
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.6.1->pandas) (1.11.0)
Installing collected packages: pytz, python-dateutil, pandas, llvmlite, numba
Successfully installed llvmlite-0.30.0 numba-0.46.0 pandas-0.25.3 python-dateutil-2.8.1 pytz-2019.3
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
tf-docker ~ > pip install matplotlib
WARNING: The directory '/project/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
WARNING: The directory '/project/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting matplotlib
  Downloading https://files.pythonhosted.org/packages/4e/11/06958a2b895a3853206dea1fb2a5b11bf044f626f90745987612af9c8f2c/matplotlib-3.1.2-cp36-cp36m-manylinux1_x86_64.whl (13.1MB)
     |████████████████████████████████| 13.1MB 1.4MB/s 
Requirement already satisfied: numpy>=1.11 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (1.17.2)
Collecting pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 (from matplotlib)
  Downloading https://files.pythonhosted.org/packages/5d/bc/1e58593167fade7b544bfe9502a26dc860940a79ab306e651e7f13be68c2/pyparsing-2.4.6-py2.py3-none-any.whl (67kB)
     |████████████████████████████████| 71kB 22.4MB/s 
Collecting cycler>=0.10 (from matplotlib)
  Downloading https://files.pythonhosted.org/packages/f7/d2/e07d3ebb2bd7af696440ce7e754c59dd546ffe1bbe732c8ab68b9c834e61/cycler-0.10.0-py2.py3-none-any.whl
Collecting kiwisolver>=1.0.1 (from matplotlib)
  Downloading https://files.pythonhosted.org/packages/f8/a1/5742b56282449b1c0968197f63eae486eca2c35dcd334bab75ad524e0de1/kiwisolver-1.1.0-cp36-cp36m-manylinux1_x86_64.whl (90kB)
     |████████████████████████████████| 92kB 35.4MB/s 
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (2.8.1)
Requirement already satisfied: six in /usr/lib/python3/dist-packages (from cycler>=0.10->matplotlib) (1.11.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from kiwisolver>=1.0.1->matplotlib) (41.2.0)
Installing collected packages: pyparsing, cycler, kiwisolver, matplotlib
Successfully installed cycler-0.10.0 kiwisolver-1.1.0 matplotlib-3.1.2 pyparsing-2.4.6
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
tf-docker ~ > 

In another shell:

$ tensorman save SPNET SPNET_IMAGE
sha256:ac1d4969699a942de3fa5a2d6a603095cdda6d2459f5dc6e73df935062191d70
$ tensorman =SPNET_IMAGE run --gpu bash
"docker" "run" "-u" "1000:1000" "--gpus=all" "-e" "HOME=/project" "-it" "--rm" "-v" "/home/shawley/SPNet:/project" "-w" "/project" "tensorman:SPNET_IMAGE" "bash"

________                               _______________                
___  __/__________________________________  ____/__  /________      __
__  /  _  _ \_  __ \_  ___/  __ \_  ___/_  /_   __  /_  __ \_ | /| / /
_  /   /  __/  / / /(__  )/ /_/ /  /   _  __/   _  / / /_/ /_ |/ |/ / 
/_/    \___//_/ /_//____/ \____//_/    /_/      /_/  \____/____/|__/

You are running this container as user with ID 1000 and group 1000,
which should map to the ID and group for your user on the Docker host. Great!

tf-docker ~ > python3
Python 3.6.8 (default, Aug 20 2019, 17:12:48) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'cv2'
>>> 

...? Any idea why this isn't working?

mmstick commented 4 years ago

Make sure you're using pip3 for Python3

drscotthawley commented 4 years ago

Thanks. I solved the problem - it was twofold! I'll outline the solution here for anyone who comes later:

  1. There was a conflict with where opencv was getting placed. For creation of the image, the appropriate place is the root python installation, not the user's .local/ path. So, I uninstalled opencv (removing the one in ~/.local/) and reinstalled it (placing it in /usr/local).
  2. The issue with libSM.so.6 came back, but that was fixed (within the tensorman image creation) by running
    apt-get install -y libsm6 libxext6 libxrender-dev

    (found that solution here).

After that, one can run python within the image-creation and check that opencv now imports properly.

Then, one can save that container as an image and all's well.

Thanks Michael! I'll close with this comment.

jeroavf commented 3 years ago

Hi , This thread helped me to solve the same problem I was having with my installation of tensorman in a Pop_OS 20.04. To have opencv fully functional I had to install "python3-opencv" as the line below:

apt-get install -y libsm6 libxext6 libxrender-dev python3-opencv

Thanks a lot for the guidance.