Installing on windows

[nutte2]

HI! Thank for a very nice book. When I try to install on a Win10 machine I get some errors when trying the requirements.txt..

a) jupyter notebook is missing. Needed to do python3 -m pip install jupyter

b) i need to change pip3 to pip and python3 to only python (I only have python 3.6.8, no 2.7)

c) tfx fails to install (Could not find a version that satisfies the requirement ml-metadata<0.15,>=0.14):

Collecting future
  Using cached https://files.pythonhosted.org/packages/3f/bf/57733d44afd0cf67580658507bd11d3ec629612d5e0e432beb4b8f6fbb04/future-0.18.1.tar.gz
Collecting dill
  Using cached https://files.pythonhosted.org/packages/c7/11/345f3173809cea7f1a193bfbf02403fff250a3360e0e118a1630985e547d/dill-0.3.1.1.tar.gz
Collecting tensorflow-metadata
  Using cached https://files.pythonhosted.org/packages/4e/c2/e4ed82a725c9f8160a0ed73f0511773be9f76343def86f6f47121f0e8430/tensorflow_metadata-0.15.0-py2.py3-none-any.whl
Collecting psutil
  Using cached https://files.pythonhosted.org/packages/86/91/f15a3aae2af13f008ed95e02292d1a2e84615ff42b7203357c1c0bbe0651/psutil-5.6.3-cp36-cp36m-win_amd64.whl
Requirement already satisfied, skipping upgrade: attrs in c:\users\kalle\venvs\my_env\lib\site-packages (from tensorflow-datasets==1.2.0->-r requirements.txt (line 52)) (19.3.0)
ERROR: Could not find a version that satisfies the requirement ml-metadata<0.15,>=0.14 (from tfx==0.14.0->-r requirements.txt (line 57)) (from versions: 0.12.0.dev0, 0.13.0.dev0, 0.13.1.dev0)
ERROR: No matching distribution found for ml-metadata<0.15,>=0.14 (from tfx==0.14.0->-r requirements.txt (line 57))

(my_env) C:\Users\Kalle\venvs\my_env\handson-ml2-master>

d) some other error (I am using TF 2.0.0 GPU) but some TF 1.14 error?)

  Building wheel for promise (setup.py) ... done
  Created wheel for promise: filename=promise-2.2.1-cp36-none-any.whl size=21294 sha256=27bd5f0ea45972456dae5e656e0a57938455f326249db262f323a750d2a77fe2
  Stored in directory: C:\Users\Kalle\AppData\Local\pip\Cache\wheels\92\84\9f\75e2235effae0e1c5a5c0626a503e532bbffcb7e79e672b606
  Building wheel for googleapis-common-protos (setup.py) ... done
  Created wheel for googleapis-common-protos: filename=googleapis_common_protos-1.6.0-cp36-none-any.whl size=77586 sha256=e6806e2edee25b1590cdc111a6cc456ecb792bbed3473faf14f107885fe345b7
  Stored in directory: C:\Users\Kalle\AppData\Local\pip\Cache\wheels\9e\3d\a2\1bec8bb7db80ab3216dbc33092bb7ccd0debfb8ba42b5668d5
Successfully built nltk opt-einsum gast absl-py future dill promise googleapis-common-protos
ERROR: tensorflow 1.14.0 has requirement tensorboard<1.15.0,>=1.14.0, but you'll have tensorboard 2.0.0 which is incompatible.
ERROR: tensorflow 1.14.0 has requirement tensorflow-estimator<1.15.0rc0,>=1.14.0rc0, but you'll have tensorflow-estimator 2.0.1 which is incompatible.
Installing collected packages: cycler, kiwisolver, numpy, pyparsing, matplotlib, pytz, pandas, scipy, scikit-learn, xgbo

Maybe you/someone have some ideas of what is wrong? Best regards / Kalle Prorok, Umeå, Sweden (teacher with 166 students on our Deep Learnig course)

[ageron]

Hi @nutte2 ,

Thanks a lot for your very useful feedback, and I'm sincerely sorry about the issues you ran into, I don't use Windows so unfortunately I could not test the installation instructions, and I haven't received much feedback from Windows users, so I assumed everything worked fine. :/

Regarding Jupyter, the requirements.txt file includes jupyter==1.0.0 so I don't understand why it was not installed when you ran pip install -r requirements.txt. Perhaps it's because pip failed because of other package failures, before jupyter could be installed?

Regarding pip and python versus pip3 and python3, it's a Windows-specific quirk that I wasn't aware of. I'll probably add a note in the book. To avoid this kind of problems, in the 2nd edition I use python3 -m pip install instead of pip3 install. The python3 still needs to be replaced with python though. And unfortunately on Debian, the pip module is not installed along with python (although it's supposed to be part of Python), you need to run apt install python3-pip, it's really quite annoying that there are so many quirks depending on the platform, it makes writing installation instructions difficult and brittle.

Regarding TFX, yes it is currently limited to MacOSX and Linux, unfortunately. I was expecting a Windows version to be out by the time my book was released, which is why this limitation is not mentioned in the book. I'll add a note.

Regarding the fact that you get a TF 1.14 error although you installed TF-GPU 2.0, this is probably because some other package you installed (probably tfx) depends on TF 1.14, so it tries to downgrade to TF 1.14 and fails. It's a bit of a mess right now, hopefully all these packages will be consistent soon.

In the meantime, perhaps the simplest option would be to install Docker and pull the ageron/handon-ml2 image that I built recently. It has all the required packages installed:

docker pull ageron/handson-ml2

I hope this helps.

[nutte2]

Thank you dear Ageron for the quick answer. Windows is really a mess for Deep Learning but many people use it so I try to make it work; I am waiting for the new Windows Subsystem for Linux (WSL2) which might solve some issues but is only available for Windows insider (betatesters) people, hopefully it will arrive during spring. (the current WSL is slow; emulating, and don't support GPU and running Ubuntu under virtual machine Hyper-V (requiring Windows 10 Pro/Edu and a Hyper-thredable CPU) also don't support GPU, neither Docker supports GPU so not very useful/effective although it works, sorry.

Running dual boot Windows+Ubuntu is a possiblity but installing with Win 10 is difficult due to the new boot security system risking failure and lost windows partitions requiring backup and often reinstall..

The Jupyter fail was strange; I removed TFX but still failed but easy to solve with an extra pip.

OpenAI Gym will be another problem; it can run on a Windows machine but with a lot of "extras"..(googling will get answers).

(Ana)Conda is maybe a possiblity but it didn't support TF2.0 some weeks ago, maybe now.

You are lucky to not use Windows ;). Maybe colab or similar is the best solution currently, Thank you for the help so far and please have a nice week!

[ageron]

Thanks for your thorough answer, it's really interesting and hopefully will help other readers.

Regarding OpenAI Gym, the main problem was getting the Atari dependencies to work on Windows, but as far as I know this was fixed recently. Did you give it a try?

Regarding Anaconda, that's what I currently use. It does not include conda packages for TF 2.0, as you mention, but conda and pip work pretty well together: I install as much as I can using conda, then at the end I install the few extra pip packages I need, including TensorFlow and all the TensorFlow-related packages.

Another option you may consider is using Colab. It's free and basically gives you a nice Jupyter-like environment running on free VMs on Google Cloud, even with free GPUs and TPUs. For courses, they're ideal (in fact I'm giving a TFX tutorial next week at TF World 100% running on Colab). If you need more resources, you can get a VM on Google Cloud (or any other cloud platform you want). These VMs cost you almost zero when you don't use them (assuming you turn them off): all you pay is the disk space, and that's pretty cheap. Whenever you need to train a model, you can fire up the VM and off you go. I did that while I was writing the book, as it gave me a lot of flexibility: if I just needed a simple VM with perhaps 1 GPU, I would use Colab. For anything bigger, I would use a Google Cloud VM with several GPU cards.

Hope this helps.

[codingbandit]

While I'm not as far in the book (yet) - this is how I setup my environment on Windows 10 http://codingbandit.com/blog/2019/12/20/machine-learning-on-windows-installing-tooling-and-setting-up-an-environment/

[maxlataci]

Setup Win 10 (no CONDA needed) Python Win10.docx

[vasili111]

You can install Anaconda and it will install IDEs and many libraries that are needed. Installation tutorial: https://www.youtube.com/watch?v=uOwCiZKj2rg

[ageron]

I recently tested the whole installation process on Windows, it's wasn't too bad: download and install git and Miniconda (or Anaconda if you prefer), then clone this repository, cd to the handson-ml2 directory and run conda env create -f environment.yml. See the detailed commands here. Hope this helps.

[twVolc]

Just to say that this installation now fails on Windows. I've followed the installation instructions but pip seems to hit an error with gym.

          raise InvalidRequirement(str(e)) from e
      wheel.vendored.packaging.requirements.InvalidRequirement: Expected end or semicolon (after version specifier)
          opencv-python>=3.
                       ~~~^
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for gym
ERROR: Could not build wheels for gym, which is required to install pyproject.toml-based projects

failed

CondaEnvException: Pip failed

There's more to the traceback than this but just to be concise I've left it out here.

Going to try moving to ml3 which will hopefully install - as I have the 2nd edition of the book I'm hoping the ml3 code and installation will be relatively backwards compatible...?

[puppysdevotion]

Just to say that this installation now fails on Windows. I've followed the installation instructions but pip seems to hit an error with gym.

          raise InvalidRequirement(str(e)) from e
      wheel.vendored.packaging.requirements.InvalidRequirement: Expected end or semicolon (after version specifier)
          opencv-python>=3.
                       ~~~^
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for gym
ERROR: Could not build wheels for gym, which is required to install pyproject.toml-based projects

failed

CondaEnvException: Pip failed

There's more to the traceback than this but just to be concise I've left it out here.

Going to try moving to ml3 which will hopefully install - as I have the 2nd edition of the book I'm hoping the ml3 code and installation will be relatively backwards compatible...?

i meet the same problem .and i find that other parkages can download successfully. i would appreciate if if u can share your solution with me

[twVolc]

Not really a great solution but I just installed the ML3 environment for the latest version of the book instead, and this environment installed successfully. I haven't worked through too much since then so I don't know if absolutely every package will be compatible with ML2 exercises, but worst-case scenario you can work through the ML3 exercises instead (I'm going to guess/hope most of the core things in the book haven't changed between the editions).

ML3 can be found here: https://github.com/ageron/handson-ml3

[puppysdevotion]

这并不是一个很好的解决方案，但我只是安装了本书最新版本的 ML3 环境，并且该环境安装成功。从那时起我就没有做过太多的工作，所以我不知道是否每个包都与 ML2 练习兼容，但最坏的情况是你可以通过 ML3 练习来代替（我是我猜测/希望书中的大部分核心内容在各个版本之间都没有改变）。

ML3 可以在这里找到：https://github.com/ageron/handson-ml3

thank you for your warm-hearted and timely reply!!