search

pop-os / tensorman

Utility for easy management of Tensorflow containers
GNU General Public License v3.0
206 stars 16 forks source link

How to add python packages to the docker image? #6

Closed phobrain closed 4 years ago

phobrain commented 4 years ago

From a quick look at docker docs, it looks like you have to build a new image with numpy etc? That must be wrong.

File "./p2.py", line 84, in import matplotlib ModuleNotFoundError: No module named 'matplotlib'

I assume all this is in the Docker image?

exec: /usr/bin/python3 path: ['/project', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages']

phobrain commented 4 years ago

Installing collected packages: kiwisolver, pyparsing, cycler, python-dateutil, matplotlib ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/local/lib/python3.6/dist-packages/kiwisolver.cpython-36m-x86_64-linux-gnu.so' Consider using the --user option or check the permissions.

$ tensorman run --gpu --python3 -- pip3 install matplotlib --user ... Installing collected packages: python-dateutil, kiwisolver, cycler, pyparsing, matplotlib ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/.local' Check the permissions.

WARNING: You are using pip version 19.2.3, however version 19.3.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command.

mmstick commented 4 years ago

I'm about to add support for this. The gist is that you need to run the container as root, install the packages you want, then docker commit <IMAGE> <NEW_NAME> to save the image, and use that image the next time you want that configuration.

phobrain commented 4 years ago

It'd be neat to be able to just list python version and all the packages and the final name of the image when setting up originally. Seems cumbersome to build a new image as one hits each new import. :-)

mmstick commented 4 years ago

It is possible to install dependencies locally (not system-wide), either on the host, or in the container, and then use those, but if you want those dependencies to be in the image on startup, then creating an image from an image is our best option.

I'm working on a solution for tensorman that makes this easier to maintain for handling tensorflow containers specifically. I should have something ready for testing by the end of the day.

mmstick commented 4 years ago

The new image-creation branch contains the ability to automate image creation and integration with tensorman.

mourgos commented 4 years ago

Will this be backported to 19.10? It seems too important of a feature to be missing.

mmstick commented 4 years ago

It's supported in https://github.com/pop-os/tensorman/pull/7. The updated README in that PR describes how to use it.

WatchMkr commented 4 years ago

The new feature should be merged soon. It's currently waiting for an engineering and QA review.