Pip installs don't always work without sudo

[RobinL]

I was not able to install lxml without sudo permissions (which only I have).

We probably need to spend some time thinking about how to install libs. For instance, do we want users using conda?

jovyan@jupyter-lab-robinl-jupyt-8b58896b6-bvb7w:~/Python_projects/platform_demos$ pip install lxml
Collecting lxml
Exception:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/opt/conda/lib/python3.6/site-packages/pip/commands/install.py", line 335, in run
    wb.build(autobuilding=True)
  File "/opt/conda/lib/python3.6/site-packages/pip/wheel.py", line 756, in build
    self.requirement_set.prepare_files(self.finder)
  File "/opt/conda/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/opt/conda/lib/python3.6/site-packages/pip/req/req_set.py", line 620, in _prepare_file
    session=self.session, hashes=hashes)
  File "/opt/conda/lib/python3.6/site-packages/pip/download.py", line 821, in unpack_url
    hashes=hashes
  File "/opt/conda/lib/python3.6/site-packages/pip/download.py", line 659, in unpack_http_url
    hashes)
  File "/opt/conda/lib/python3.6/site-packages/pip/download.py", line 853, in _download_http_url
    stream=True,
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pip/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/cachecontrol/adapter.py", line 37, in send
    cached_response = self.controller.cached_request(request)
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/cachecontrol/controller.py", line 111, in cached_request
    resp = self.serializer.loads(request, cache_data)
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/cachecontrol/serialize.py", line 114, in loads
    return getattr(self, "_loads_v{0}".format(ver))(request, data)
  File "/opt/conda/lib/python3.6/site-packages/pip/_vendor/cachecontrol/serialize.py", line 176, in _loads_v2
    cached = json.loads(zlib.decompress(data).decode("utf8"))
zlib.error: Error -3 while decompressing data: invalid literal/length code
You are using pip version 9.0.3, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
jovyan@jupyter-lab-robinl-jupyt-8b58896b6-bvb7w:~/Python_projects/platform_demos$ sudo pip install lxml
The directory '/home/jovyan/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/jovyan/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting lxml
  Downloading https://files.pythonhosted.org/packages/03/a4/9eea8035fc7c7670e5eab97f34ff2ef0ddd78a491bf96df5accedb0e63f5/lxml-4.2.5-cp36-cp36m-manylinux1_x86_64.whl (5.8MB)
    100% |████████████████████████████████| 5.8MB 109kB/s
Installing collected packages: lxml
Successfully installed lxml-4.2.5
You are using pip version 9.0.3, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
jovyan@jupyter-lab-robinl-jupyt-8b58896b6-bvb7w:~/Python_projects/platform_demos$

[davidread]

Weird, it installs fine for me without sudo:

jovyan@jupyter-lab-davidread-ju-d869bf94-ch4s5:~$ pip install lxml
Collecting lxml
  Downloading https://files.pythonhosted.org/packages/03/a4/9eea8035fc7c7670e5eab97f34ff2ef0ddd78a491bf96df5accedb0e63f5/lxml-4.2.5-cp36-cp36m-manylinux1_x86_64.whl (5.8MB)
    100% |████████████████████████████████| 5.8MB 107kB/s
Installing collected packages: lxml
Successfully installed lxml-4.2.5
You are using pip version 9.0.3, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
jovyan@jupyter-lab-davidread-ju-d869bf94-ch4s5:~$ python -c 'import lxml; print(lxml)'
<module 'lxml' from '/home/jovyan/.local/lib/python3.6/site-packages/lxml/__init__.py'>

But I have xmllib2 installed by conda already (in /opt/conda):

$ conda list |grep xml
libxml2                   2.9.8                h422b904_2    conda-forge

But I can't edit the install of course:

$ conda uninstall libxml2
Solving environment: failed

NotWritableError: The current user does not have write permissions to a required path.
  path: /home/jovyan/.conda/pkgs/urls.txt
  uid: 1001
  gid: 100

If you feel that permissions on this path are set incorrectly, you can manually
change them by executing

  $ sudo chown 1001:100 /home/jovyan/.conda/pkgs/urls.txt

In general, it's not advisable to use 'sudo conda'.

Could we give users sudo access? Without it conda wouldn't let them install system libraries. It seems concerning.

Ideas:

• users can create a text file listing conda packages that are installed on container start.
• users can create a dockerfile for their jupyter container, or at least create PRs that a dev can review and merge
• users can create a script that is run on container start. This is what Sagemaker does. There is an example which uses sudo here: https://aws.amazon.com/blogs/machine-learning/running-fast-ai-notebooks-with-amazon-sagemaker/ Maybe they are less concerned about security because it is running on an EC2 VM, rather than a pod on a multi-user VM.

[davidread]

Robin suggested we focus on making sure our standard image includes all the common python/R libraries with any system packages installed, e.g. anaconda.

I guess there will come a situation when this is not enough. e.g. you need a more recent version of libxml than the default, and want to install it with sudo to be the default system package, so that the R/python library uses it? For the time being an admin can do this by editing the dockerfile, but when that becomes unsustainable then we can see about letting users manage it.

[davidread]

@RobinL The error you got originally is: zlib.error: Error -3 while decompressing data: invalid literal/length code Doesn't seem like a permissions error. Could that simply be a one-off network error?

[davidread]

Robin said this wasn't a one off. I can reproduce the error by execing into his jupyter container.

Ah, I've solved it now. In Robin's container I managed to install lxml fine by telling pip not to use its cached copy of the .whl:

pip install lxml --no-cache-dir -v

The cached copy must have been corrupt - an empty file I suspect.