old html5lib makes pip dysfunctional

[alanpurple]

html5lib dependency should be updated to 1.0b10

windows 10 x64 Anaconda 5.0.0 with python 3.6.2 install tensorflow-gpu -> tensorboard -> html5lib downgraded -> pip dysfunctional

[chihuahua]

Interesting - in what way does pip become dysfunctional?

[alanpurple]

@chihuahua TypeError: parse() got an unexpected keyword argument 'transport_encoding'

similar to https://github.com/Kozea/WeasyPrint/issues/353

Resolved by copying newer version of html5lib folder and overwrite

[chihuahua]

Hmm, TensorBoard already requires 1.0b8, which comes after 1.0b1. https://github.com/tensorflow/tensorboard/blob/f6e82c446f963db689b646cbde0a39f1d201ca47/tensorboard/pip_package/setup.py#L34

[alanpurple]

@chihuahua my mistake, b10

[jart]

Upgrading to a newer html5lib broke our bleach dependency. We would need to upgrade Google to the latest Bleach in order to do this. What specific version of pip are you using? In what way does it become dysfunctional? What is the specific error, including traceback?

[alanpurple]

@jart

Windows PowerShell Copyright (C) 2016 Microsoft Corporation. All rights reserved.

PS C:\Users\alanp> pip -V pip 9.0.1 from C:\Anaconda3\lib\site-packages (python 3.6) PS C:\Users\alanp> conda upgrade pip Fetching package metadata ............... Solving package specifications: .

All requested packages already installed.

packages in environment at C:\Anaconda3:

# pip 9.0.1 py36hadba87b_3 PS C:\Users\alanp> conda upgrade setuptools Fetching package metadata ............... Solving package specifications: .

All requested packages already installed.

packages in environment at C:\Anaconda3:

# setuptools 36.5.0 py36h65f9e6e_0 PS C:\Users\alanp> conda install bleach==1.5.0 Fetching package metadata ............... Solving package specifications: .

Package plan for installation in environment C:\Anaconda3:

The following packages will be DOWNGRADED:

bleach:   2.0.0-py36h0a7e3d6_0       --> 1.5.0-py36_0
html5lib: 0.999999999-py36ha09b1f3_0 --> 0.9999999-py36_0

Proceed ([y]/n)?

html5lib-0.999 100% |###############################| Time: 0:00:00 11.67 MB/s bleach-1.5.0-p 100% |###############################| Time: 0:00:00 0.00 B/s PS C:\Users\alanp> pip -V pip 9.0.1 from C:\Anaconda3\lib\site-packages (python 3.6) PS C:\Users\alanp> pip install sqlalchemy==1.1.13 Collecting sqlalchemy==1.1.13 Exception: Traceback (most recent call last): File "C:\Anaconda3\lib\site-packages\pip\basecommand.py", line 215, in main status = self.run(options, args) File "C:\Anaconda3\lib\site-packages\pip\commands\install.py", line 335, in run wb.build(autobuilding=True) File "C:\Anaconda3\lib\site-packages\pip\wheel.py", line 749, in build self.requirement_set.prepare_files(self.finder) File "C:\Anaconda3\lib\site-packages\pip\req\req_set.py", line 380, in prepare_files ignore_dependencies=self.ignore_dependencies)) File "C:\Anaconda3\lib\site-packages\pip\req\req_set.py", line 554, in _prepare_file require_hashes File "C:\Anaconda3\lib\site-packages\pip\req\req_install.py", line 278, in populate_link self.link = finder.find_requirement(self, upgrade) File "C:\Anaconda3\lib\site-packages\pip\index.py", line 465, in find_requirement all_candidates = self.find_all_candidates(req.name) File "C:\Anaconda3\lib\site-packages\pip\index.py", line 423, in find_all_candidates for page in self._get_pages(url_locations, project_name): File "C:\Anaconda3\lib\site-packages\pip\index.py", line 568, in _get_pages page = self._get_page(location) File "C:\Anaconda3\lib\site-packages\pip\index.py", line 683, in _get_page return HTMLPage.get_page(link, session=self.session) File "C:\Anaconda3\lib\site-packages\pip\index.py", line 811, in get_page inst = cls(resp.content, resp.url, resp.headers) File "C:\Anaconda3\lib\site-packages\pip\index.py", line 731, in init namespaceHTMLElements=False, TypeError: parse() got an unexpected keyword argument 'transport_encoding' PS C:\Users\alanp> conda upgrade bleach Fetching package metadata ............... Solving package specifications: .

Package plan for installation in environment C:\Anaconda3:

The following packages will be UPDATED:

bleach:   1.5.0-py36_0     --> 2.0.0-py36h0a7e3d6_0
html5lib: 0.9999999-py36_0 --> 0.999999999-py36ha09b1f3_0

Proceed ([y]/n)?

PS C:\Users\alanp> pip install sqlalchemy==1.1.13 Collecting sqlalchemy==1.1.13 Installing collected packages: sqlalchemy Found existing installation: SQLAlchemy 1.1.14 Uninstalling SQLAlchemy-1.1.14: Successfully uninstalled SQLAlchemy-1.1.14 Successfully installed sqlalchemy-1.1.13 PS C:\Users\alanp> pip install sqlalchemy==1.1.14 Collecting sqlalchemy==1.1.14 Installing collected packages: sqlalchemy Found existing installation: SQLAlchemy 1.1.13 Uninstalling SQLAlchemy-1.1.13: Successfully uninstalled SQLAlchemy-1.1.13 Successfully installed sqlalchemy-1.1.14

[jart]

Maybe file an issue upstream with pip or probably conda? Having an old version of a package shouldn't break a package manager.

[alanpurple]

@jart

conda has nothing to deal with this I just use conda for reproduce this situation safely(for safe and easy back)

pip install tensorflow-gpu->tensorboard->bleach1.5.0->html5lib b8 has same problem

[jart]

If what you're saying is the case, then all versions of pip released in past eleven months (since https://github.com/pypa/pip/pull/4038) will break when TensorFlow is installed.

@dstufft would it be possible to make pip more bullet proof in cases where old versions of html5lib are installed?

[dstufft]

Uh, something is wrong because pip doesn't normally depend on anything, it bundles all of it's dependencies internally and uses them instead unless someone has patched your version to debundle them.

[alanpurple]

@dstufft well, apparently pip use htmllib's "parse"

@jart whatever the true cause is, you guys can't use old bleach and html5lib forever ( in this nature )

It will cause another problem after python 3.6.3, 3.7.0 or Anaconde 5.x.x, etc ( possibly )

[dstufft]

We do yes, but one one that has been vendored into pip. We don't use the installed version at all, see https://github.com/pypa/pip/blob/9.0.1/pip/index.py#L33.

[henrysky]

I am also suffering from this issue.

pip install tensorflow-gpu will break my pip by installing an outdated version of html5lib You can resolve the issue temporarily by installing a newer version of html5lib wheel downloaded to your computer locally

But updating a newer version of html5libwill break tensorboard

(C:\Users\Henry\Anaconda3) C:\Users\Henry\Documents>tensorboard Traceback (most recent call last): File "c:\users\henry\anaconda3\lib\runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "c:\users\henry\anaconda3\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\Henry\Anaconda3\Scripts\tensorboard.exe__main.py", line 5, in File "c:\users\henry\anaconda3\lib\site-packages\tensorboard\main.py", line 36, in from tensorboard.plugins.audio import audio_plugin File "c:\users\henry\anaconda3\lib\site-packages\tensorboard\plugins\audio\audio_plugin.py", line 27, in from tensorboard import plugin_util File "c:\users\henry\anaconda3\lib\site-packages\tensorboard\plugin_util.py", line 21, in import bleach File "c:\users\henry\anaconda3\lib\site-packages\bleach\init__.py", line 14, in from html5lib.sanitizer import HTMLSanitizer ModuleNotFoundError: No module named 'html5lib.sanitizer'

[jart]

If pip shades its dependencies (e.g. html5lib becomes pip._vendor.html5lib) then the only likely explanation is the conda command is wrapping pip in such a way that the shading doesn't happen.

@kalefranz There have been multiple reports that installing TensorFlow with Conda will break the Conda installation, most likely due to pip's dependencies not being shaded. What can we do to help our friends?

[alanpurple]

update bleach version dependency to 2.1.1 and htmllib5 to 1.0b10 will resolve this issue

[marcotama]

I can confirm that changing the requirements in metadata.json inside the .whl of tensorflow-tensorboard as suggested by @alanpurple no longer breaks the installation. I am on Windows 10, Python 3.6.2, Anaconda, tensorflow-tensorboard==1.3.0 and tensorflow==1.3.0.

[wang1943]

I have experienced a similar issue with tensorboard recently that tensorboard broken when install tensorflow in conda #630 . Does anyone know if there is a timeline for official fix? I'm new to tensorflow and currently take a class for deep learning (tensorboard is kinda important to illustrate concepts in the class). I can work with tensorflow on Linux environment but it would be much nicer if the windows installation can be fixed.

[marcotama]

Hi @wang1943 , I resorted to using pip install tensorflow==1.2.0. Notice that in order to have 1.2.0 work with the GPU, I had to install cuDNN 5.1 (not 5.0!), which I was able to find here: https://developer.nvidia.com/rdp/cudnn-download

If you broke the pip installation, you can fix it by downloading the whl file of html5lib from https://pypi.python.org/pypi/html5lib and then running pip install <path-to-whl-file>

[wang1943]

@nazgul17 Cool, thanks! That's work for me.

[straygar]

@marcotama I'm confused. How are you supposed to pip install anything if the pip installation on the environment is broken, throwing an exception every time you try to do anything?

[marcotama]

@straygar pip install <file> installs a package from a local file, thereby working around the broken html5lib.

[gwerbin]

@jart this does sound like an upstream issue with Conda, I never had this problem until I switched from Pip to Conda.

[kalefranz]

This is only happening on windows, correct? Has anyone seen this on an OS other than windows?

[msarahan]

As an explanation of what's going on here, Anaconda needed to un-vendor pip due to concerns about Windows path length limit (260 characters). The vendored packages were exceeding that limit in our tests, causing conda installations of pip to fail. IIRC, it was the vendoring of vendored deps that was the most painful: https://github.com/pypa/pip/tree/9.0.1/pip/_vendor/requests/packages - this appears to be fixed in master, though, so we won't need to un-vendor future pip versions.

We're not sure exactly what the proper fix is yet for current pip, but we'll work on a fix ASAP. Any advice on how to un-vendor better is welcome.

[jart]

@msarahan I'm so happy you're here. If path length is an issue on Windows, then one thing you might be able to do is just stuff pip in a zip and add it to sys.path. As in cd lib/site-packages && zip -r pip.zip pip and then sys.path.append('pip.zip'); from pip import main; main().

I just tried it on my workstation and got an SSL error when I ran pip install, so there might be a couple lines in there that need to be monkey-patched, like possibly to load a pem file or something. But it should be doable.

The cherry on top, if you do this, is pip will have a better cold start time with Conda than anywhere else. Files within a zip tend to occupy contiguous memory, so fewer seeks and better readahead. Also way less system calls because stat() calls end up being userspace hash map lookups on the zip directory loaded from the end of the file into memory. We've actually been thinking about doing this exact same thing with TensorFlow, so it won't take a half a darn second to import.

[elipeters]

Windows 10 fix with anaconda conda install -c anaconda html5lib

[ghost]

@elipeters solution has worked for me

[msarahan]

We also have a new build of pip available for testing. This new version restores vendoring in pip, but uses the master branch of pip for the vendorings, which are an updated, unvendored requests, which makes the path length issue go away. Unfortunately, it probably missed our bugfix release for the 5.0.0.1 version of Anaconda. You can test it with:

conda install -c c3i_test2 pip

[back2yes]

met same bug on windows 7

[alanpurple]

tensorflowers!!! please stop making html5lib downgrade everytime

why is this necessary to force to downgrade?

[msarahan]

Our new pip version is available by default now. We hope this fixes the issue on the anaconda side.

conda update pip

[alanpurple]

@msarahan thanks, I'll try that one

[BenjaminWegener]

https://github.com/SerpentAI/SerpentAI/issues/96#issuecomment-336203943

[halfhorst]

The pip upgrade solved this for me. Thanks!

[fatih-iver]

pip install html5lib==1.0b10 solved my problem. Thanks!

[MikeTheWatchGuy]

marcotama ! THANK YOU !

You're the only post on the net so far that has said: DO THIS I did and it fixed it... Thanks tons!

[lukehsiao]

When I include tensorboard as one of my dependiencies in my own python package, and try to python setup.py install, I'm getting:

error: html5lib 1.0b8 is installed but html5lib==0.9999999 is required by {'tensorboard'}

Note that 1.0b8 should be identical to 0.9999999 as noted in the comment. Is there an easy way for me to resolve this issue?

[anentropic]

I have problems with this outdated html5lib dependency when trying to use pipenv

https://github.com/pypa/pipenv/issues/1716