apple / tensorflow_macos

TensorFlow for macOS 11.0+ accelerated using Apple's ML Compute framework.
Other
3.66k stars 308 forks source link

Instructions to install TensorFlow in a Conda Environment #153

Open mwidjaja1 opened 3 years ago

mwidjaja1 commented 3 years ago

This is not so much an issue as opposed to a 'How To' if you'd like to install this version of Tensorflow in Conda.

Prerequisites: You must be on macOS Big Sur

If you have an Apple Silicon Mac, this is a freebie, you're already on Big Sur. If you're on an Intel Mac, the Intel versions of TensorFlow are Big Sur only.

Sanity Check before Proceeding: To ensure you're on the right version of macOS, run sw_vers -productVersion in your terminal. If it's not version 11.##, you're not on Big Sur and must upgrade to it from the macOS App Store.

Prerequisites: Install XCode Command Line Tools

Install Xcode Command Line tools if you haven't. To do so, run this in your terminal: xcode-select --install

Sanity Check before Proceeding: To ensure installation worked, run which xcrun in your terminal and you should get a path like /usr/bin/xcrun. If you haven't, you did not install it correctly.

Prerequisites: Install Miniforge

Where to download Miniforge from Miniforge, is a 'lightweight' Python interpreter that has full access to the Conda ecosystem. You can download Miniforge from https://github.com/conda-forge/miniforge#miniforge3. You can use Anaconda if you're on Intel, but note that this guide will be written from the perspective of using miniforge.

Sanity Check before Proceeding:

If any of those sanity checks failed, you must redo this section. Please ensure that you downloaded the correct Miniforge for your system architecture and installed it. If you did all that, set your environment paths to Miniforge's Python Installation. To do that, you need to figure out where conda was installed to (it's probably ~/miniforge3/condabin/conda) and then run ~/miniforge3/condabin/conda init in your terminal.

Apple Silicon Only Warning: You CANNOT use Anaconda

This warning only applies to Apple Silicon Macs. Anaconda comes with many Python packages included, some of which are not Apple Silicon (i.e. ARM) compatible and thus Anaconda is not ARM compatible. You can use Anaconda if you're using an Intel Mac though.

If you were planning to use Anaconda on ARM, please scroll back up and install Miniforge. Miniforge has Conda, which means you can install many of the packages you want such as Pandas, Scipy, and Numpy -- unlike Anaconda, you just have to do the install manually by running conda install mypackagenamehere.

Intel Only Warning: Python Bugs in Big Sur

This warning only apply to Intel Macs. For Intel, both Anaconda and MiniForge have a Python Bug which prevents you from running Python correctly in some instances on macOS Big Sur. Until the Python community fixes this, each time prior to loading Python, you must run export SYSTEM_VERSION_COMPAT=0. You could also add this to your .bash_profile or other shell environment file if you have one, to do this automatically for you.

Installing TensorFlow

Attached to this Issue is a YAML file which will help you create a Conda Environment with TensorFlow, along with all the prerequisites you need from the ARM conda-forge channel.

  1. Download environment.yml, which contains the instructions to create a Python environment with the dependencies you need -- we'll install TensorFlow afterwards. Some browsers insist on adding .txt to the end of the file -- do not let your browser do that. [thanks to @isuruf for streamlining this file to be all Conda]
  2. In your terminal run this command, replacing the uppercase variables with the path to your environment.yml file and your desired name for this environment: conda env create --file=PATH_TO_ENVIRONMENT.YML --name=YOUR_ENV_NAME_HERE.
  3. Activate that environment by running this command, replacing the uppercase variable with your environment's name: conda activate YOUR_ENV_NAME_HERE
  4. Pip install the TensorFlow wheels by running the commands below. By the way, the URLs for the TensorFlow wheel files came from the Releases page, so you can swap these wheel files out with a prior version of TensorFlow as needed.

For X86 as of 03/11/2021: Thanks to @edwin-yan for the updated commands

pip install --upgrade --force --no-dependencies https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_x86_64.whl https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_addons_macos-0.1a3-cp38-cp38-macosx_11_0_x86_64.whl 

For Apple Silicon as of 03/11/2021:

pip install --upgrade --force --no-dependencies https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_addons_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl
  1. Finally, give it a spin. Run python and try importing tensorflow.

Example Commands

In this below example, I'm installing & running the ARM version of tensorflow from an environment I've named test. The yml file is placed in the same directory I'm running this command from, which is my home directory (i.e. ~)

conda env create --file=environment.yml --name=test
conda activate test
pip install --upgrade --force --no-dependencies https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha2/tensorflow_addons_macos-0.1a2-cp38-cp38-macosx_11_0_arm64.whl https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha2/tensorflow_macos-0.1a2-cp38-cp38-macosx_11_0_arm64.whl
python
import tensorflow
Screen Shot 2021-02-02 at 10 19 26 PM

Troubleshooting for importing TensorFlow

Troubleshooting for setting up TensorFlow

thomastiotto commented 3 years ago

I couldn't install the wheels without renaming them to read "...-macosx_1016..." instead of "...-macosx_110..."

mwidjaja1 commented 3 years ago

@Tioz90 I didn't have that problem. Could you tell me what your Mac's version is? That's Apple (in the Menu Bar) > About my Mac. Mine is Version 11.2.

thomastiotto commented 3 years ago

@mwidjaja1 MacOs 11.1

mwidjaja1 commented 3 years ago

@Tioz90 You may want to create a new issue then about this. I'm not on the team working on this, I just wrote up these instructions as a 'side contributor' with their help.

thomastiotto commented 3 years ago

I think it's a known issue with pip

mwidjaja1 commented 3 years ago

That's an interesting thought @Tioz90 -- My pip version is 20.2.4 which I checked by running pip --version. If yours isn't that version, could you get that version and try again?

thomastiotto commented 3 years ago

Mine is 21.0.1 and it still does not work..

mwidjaja1 commented 3 years ago

@Tioz90 I cannot reproduce your issue. I've upgraded to the same pip version you had, trashed my environment, and was able to successfully rebuild it. I have a hunch somewhere in your Python installation, you're accidentally linking to some other Python that you did not intend to use but I wouldn't know how to begin figuring that out. Glad you got it to work by renaming it though

hoytak commented 3 years ago

There's a few things that could make this issue a mess. I haven't found versions of pip later than 20.2.4 to work here; version 20.3 and later actually broke some of the logic. Also, the logic inside of pip is duplicated across setuptools and packaging, both of which are baked into the core python distribution; upgrading these might help, but your mileage may vary.

Stanfording commented 3 years ago

Does it mean that installing TensorFlow in a Conda Environment allows m1 chip to use tensorflow with SciPy and dependent packages?

thomastiotto commented 3 years ago

Yes, SciPy works for me

Thomas Tiotto On 4 Feb 2021, 10:09 +0100, Stanfording notifications@github.com, wrote:

Does it mean that installing TensorFlow in a Conda Environment allows m1 chip to use tensorflow with scipy? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

gutsyy commented 3 years ago

I got this Error ERROR: tensorflow_macos-0.1a2-cp38-cp38-macosx_11_0_x86_64.whl is not a supported wheel on this platform. This is my System Information:

I'm Confused... 

Run python -m pip debug --verboseand find pip not support cp38-cp38-macosx_11_0_x86_64 only support cp38-cp38-macosx_10_16_x86_64

Indeed: Python 3.8 from the system:

/usr/bin/python3 -c "import platform; print(platform.platform())"
macOS-11.0.1-x86_64-i386-64bit

Python 3.9 installed with homebrew:

/usr/local/bin/python3 -c "import platform; print(platform.platform())"
macOS-11.0.1-x86_64-i386-64bit

Python 3.8 from a conda-forge env:

/Users/ogrisel/miniforge3/envs/py38/bin/python -c "import platform; print(platform.platform())"
macOS-10.16-x86_64-i386-64bit

So conda-forge's Python is indeed pretending to be installed on a fake 10.16 version of macOS to bypass this problem with pip 20.2.4.

mwidjaja1 commented 3 years ago

@Tioz90, I think @gyuannnn figured it out. I get your exact error on Anaconda on Intel too. Interestingly enough on an identical macOS version on an Apple Silicon Mac, it actually is 11_0. So it looks like it's all Conda Pythons (since I tried on Anaconda and you tried Miniforge) on Big Sur Intel, the platform is being saved as 10_16 rather than 11_0.

Not sure how to report this to the Miniforge folks but I can poke around unless you have an idea where the problem might be. In the meanwhile, can you try downloading the wheel files and renaming it so instead of having 11_0 in the two wheel file names, it says 10_16 instead and try pip installing those? If that works, I'll update the Readme for this workaround.

@hoytak tagging you for situational awareness that we figured out the issue.

EDIT: Nevermind I see you reported it already. https://github.com/conda-forge/miniforge/issues/105. Not sure if this is the right place admittedly (it could be Conda's fault) but let's see if they know where to direct us to.

gutsyy commented 3 years ago

@Tioz90, I think @gyuannnn figured it out. I get your exact error on Anaconda on Intel too. Interestingly enough on an identical macOS version on an Apple Silicon Mac, it actually is 11_0. So it looks like it's all Conda Pythons (since I tried on Anaconda and you tried Miniforge) on Big Sur Intel, the platform is being saved as 10_16 rather than 11_0.

Not sure how to report this to the Miniforge folks but I can poke around unless you have an idea where the problem might be. In the meanwhile, can you try downloading the wheel files and renaming it so instead of having 11_0 in the two wheel file names, it says 10_16 instead and try pip installing those? If that works, I'll update the Readme for this workaround.

@hoytak tagging you for situational awareness that we figured out the issue.

EDIT: Nevermind I see you reported it already. conda-forge/miniforge#105. Not sure if this is the right place admittedly (it could be Conda's fault) but let's see if they know where to direct us to.

Downloaded and renamed it to 1016 works fine.

gutsyy commented 3 years ago

@mwidjaja1 according to conda-forge/miniforge#105 export SYSTEM_VERSION_COMPAT=0 before running python (on env) can set it to 110

thomastiotto commented 3 years ago

@Tioz90, I think @gyuannnn figured it out. I get your exact error on Anaconda on Intel too. Interestingly enough on an identical macOS version on an Apple Silicon Mac, it actually is 11_0. So it looks like it's all Conda Pythons (since I tried on Anaconda and you tried Miniforge) on Big Sur Intel, the platform is being saved as 10_16 rather than 11_0.

Not sure how to report this to the Miniforge folks but I can poke around unless you have an idea where the problem might be. In the meanwhile, can you try downloading the wheel files and renaming it so instead of having 11_0 in the two wheel file names, it says 10_16 instead and try pip installing those? If that works, I'll update the Readme for this workaround.

@hoytak tagging you for situational awareness that we figured out the issue.

EDIT: Nevermind I see you reported it already. conda-forge/miniforge#105. Not sure if this is the right place admittedly (it could be Conda's fault) but let's see if they know where to direct us to.

Yes, renaming the files installs them without an issue

mwidjaja1 commented 3 years ago

@gyuannnn Thanks for the heads up, I updated the documentation to reflect this temporary workaround. Python 3.8.8 sounds like it'll fix this issue fortunately.

Conversely, renaming the files may crash your Python @Tioz90 if you ever update your Python to 3.8.8 which is planned to fix this issue. https://www.python.org/downloads/release/python-387/

@hoytak A heads up for your team's awareness that this is an issue... and actually might explain some of the crazy pip commands I noticed you guys pulling off. I'm starting to wonder if some of the weird installation issues I saw, while trying to use Conda installs to setup TensorFlow's pre-reqs, were related to this whole 10_16 vs. 11_0 mishap.

icenando commented 3 years ago

How can I "point" python and pip to use their respective miniforge versions? I have my conda environment working fine, but when I enter which python and which pip I get /usr/bin/python and /usr/bin/pip respectively.

I tried installing as above but got the wheel error: ERROR: tensorflow_macos-0.1a2-cp38-cp38-macosx_11_0_x86_64.whl is not a supported wheel on this platform.

mathieuboudreau commented 3 years ago

How can I "point" python and pip to use their respective miniforge versions? I have my conda environment working fine, but when I enter which python and which pip I get /usr/bin/python and /usr/bin/pip respectively.

I tried installing as above but got the wheel error: ERROR: tensorflow_macos-0.1a2-cp38-cp38-macosx_11_0_x86_64.whl is not a supported wheel on this platform.

+1 Ran into the exact same issue today.

mwidjaja1 commented 3 years ago

@icenando & @mathieuboudreau this means Miniforge is not added to your system path. This is usually a Googling task just because everybody's terminals shell prompts are different, it'd be way beyond the scope of this post to cover every permutation.

In theory running conda init in your terminal should fix it. If it complains it cannot find conda, that means you need to figure out where you installed it to and run this command again from that directory where conda resides. If that doesn't work you'll need to identify where the Python executable that Miniforge installed is, than figure out what Shell Language your terminal is using by running echo $SHELL, and then Googling how you'd add the directory containing Python to your system paths environment variable.

icenando commented 3 years ago

@mwidjaja1 Thank you for that. conda init sorted the python and pip issues, but I'm still getting ERROR: grpcio-1.33.2-cp38-cp38-macosx_10_16_arm64.whl is not a supported wheel on this platform. (I tried renaming the file to 10_16 like others recommened).

I notice that the python version installed with miniforge is 3.9. So I tried changing the pip install line to python3.9 instead of 3.8, but I got the same wheel error.

When I try using Apple's .sh file, ERROR: Python version in specificed virtual environment /Users/nandom/miniforge3/envs/TF not 3.8. Python 3.8 required for tensorflow_macos 0.1a2. .

Any help to sort this out would be appreciated.

icenando commented 3 years ago

Ok, so removing the evironment and re-creating it, specifying python=3.8 solved it. There were other steps that I had to adapt. This post is the only thing that actually worked (I had to change the filenames in the pip installs, as they have changed since it was written): https://claytonpilat.medium.com/tutorial-tensorflow-on-an-m1-mac-using-jupyter-notebooks-and-miniforge-dbb0ef67bf90

mwidjaja1 commented 3 years ago

@icenando I'm glad you got it working though just to get feedback, I'm not sure why that article would work whereas these notes wouldn't. "Installing TensorFlow" section of this guide tells you to download environment.yml and then activate said environment. That environment is designed to give you Python 3.8 and all of the prerequisites you need.

With all due respect, your first comment implies you didn't read my instructions, because you're right miniforge installs Python 3.9 by default but my instructions literally guide you away from using it. But if you have any error messages while setting up that environment or activating it, please let me know, I'd be glad to rephrase the guide. Thanks.

icenando commented 3 years ago

@mwidjaja1 Sorry, I didn't mean to imply that your instructions are incorrect. I'm sure I'm making a lot of mistakes as I go: I'm not that experienced, and there are a lot of instructions that I follow blindly because I don't understand what they do.

I don't remember getting any error messages prior to the wheel one. But regardless, that meant that I was unable to finish the install. Not that I know what I'm talking about, but wouldn't it be good to replace conda env create --file=PATH_TO_ENVIRONMENT.YML --name=YOUR_ENV_NAME_HERE with conda env create --file=PATH_TO_ENVIRONMENT.YML --name=YOUR_ENV_NAME_HERE python=3.8, just to be sure that the correct version is in use in the newly created environment?

The troubleshooting section of that article also covered many of the errors that I was getting, as well as how to install libraries that I was unable to before (I did have a working environment in place before things went pear shaped).

Apologies again if my previous message implied that there was something wrong with your instructions.

mwidjaja1 commented 3 years ago

@icenando No worries! I've been there too -- I just have been getting more DMs about this and I'm just a 'volunteer' so I've been running around heh.

If you wouldn't mind, could you give the original command another whirl? So long as you name that Conda environment something different than what you did with your working tensorflow env, they shouldn't stomp each other out. Specifically the commands I'm asking you to run are:

conda env create --file=PATH_TO_ENVIRONMENT.YML --name=YOUR_ENV_NAME_HERE
conda activate YOUR_ENV_NAME_HERE
python --version

The YML file you're downloading is the 'recipe' that Conda will use to download everything and set Python up. Because of that, there SHOULD be no need to specify python=3.8 in that command because it's in the environment file. BUT, I could definitely be wrong so if that doesn't work, I'd be glad to look at the error some more.

icenando commented 3 years ago

@mwidjaja1 You are right, the python version is 3.8.6, so that's not what caused the wheel problem. I really don't know what else it could have been... I read somewhere that the order in which things got installed mattered, so maybe that?

mwidjaja1 commented 3 years ago

@icenando [shrugs] Who knows? Either way I'm glad you got it working somehow! Your questions led to me adding the conda init shoutout to the documentation which I think is a good add anyway, so thanks for accidentally guiding me in that way 🙂

maguileraf commented 3 years ago

I did everything suggested on here but still can't seem to be able to install it. I still get ERROR: tensorflow_macos-0.1a2-cp38-cp38-macosx_10_16_arm64.whl is not a supported wheel on this platform. Any other suggestions?

mwidjaja1 commented 3 years ago

@maguileraf can you please run all of the Sanity Checks in the guide (there were several of them) and paste those results here?

Also please try to send me a screenshot of your terminal window, running the same types of commands I ran when I posted my screenshot in the "Example Commands" section.

taisuke-ito commented 3 years ago

thanks you. I was able to run. I was able to "learn" with MNIST, but tensorflow.keras uses the pillow package to load the image.

from PIL import Image

I get an error.

maguileraf commented 3 years ago
Screen Shot 2021-02-12 at 12 27 10 PM
gosstyla commented 3 years ago

After multiple attempts all using new clean environments got up to step 5.

On import TensorFlow receive message ModuleNotFoundError: No module named 'flatbuffers' Seems error occurs running: .../lib/python3.8/site-packages/tensorflow/lite/python/util.py However conda list shows flatbuffers version: 1.12.0 build: h9f76cd9_0 channel: conda-forge

This is on a new Apple M1 Air with macOS 11.2.1

I did not create latest environment using file environment.yml Instead created only including python3.8 then did conda install separately, in order, for each item listed in environment.yml. For the few items not found by conda install I used pip install.

Previous attempts when environments were created using file environment.yml I got some of the errors mentioned above by others when running step 4. command.

Any suggestions what to try next?

mwidjaja1 commented 3 years ago

@gosstyla It's my understanding that Apple's Tensorflow is very finicky with the packages it accepts and where it comes from. Conda packages could differ from the Pip packages. My request would be for you to try it again using the environment as per the instructions. If it works, I'd appreciate it if you can file a new issue about flatbuffers in this repo. [Disclaimer: I am just a volunteer on the side who wrote these instructions, my knowledge of the state of Apple's TensorFlow is about 1%]

@taisuke-ito Please file a new issue, that's outside the scope for both this issue (installing via Conda) and my role on the project (I'm just a volunteer). That said I am aware of issues with Keras on ARM so it could be related to that -- I wouldn't be surprised if you just can't use Keras at all right now.

@maguileraf

isuruf commented 3 years ago

In conda flatbuffers is the C/C++ library. python-flatbuffers is the python library.

solie commented 3 years ago

I use mambaforge flavor instead of miniforge, so far so good.

gosstyla commented 3 years ago

mwidjaja1

I did try again - following steps 1-5 exactly including using file environment.yml. IT WORKED!

Thanks, next I'll try actually using tensorflow.

ramicaza commented 3 years ago

awesome writeup! just a heads up for those having issues with tf.keras.models.load_model: make sure to downgrade to h5py to the 2.10.0 packaged with the alpha releases (just pip install ~/path to h5py.whl). otherwise you get a 'failed to decode' error

maguileraf commented 3 years ago

I had installed the wrong Miniconda version for my laptop's architecture. pip command now works without issues

mwidjaja1 commented 3 years ago

Thanks @ramicaza, I'll add that to the ever growing troubleshooting section and credit you for it

Jinlixian commented 3 years ago

I can't download environment.yml

mwidjaja1 commented 3 years ago

I can't download environment.yml

That link still works for me, even from a computer that's not signed into my account. If that's actually a problem for you, please provide any error messages or issues you're seeing.

Jinlixian commented 3 years ago

I can't download environment.yml

That link still works for me, even from a computer that's not signed into my account. If that's actually a problem for you, please provide any error messages or issues you're seeing.

Sorry, maybe it's because of the network problem in China. Now it's solved. thank you

epignatelli commented 3 years ago

Guys, I appreciate all the effort your are putting in this pre-release installation, but many of us have problems on the Apple M1:

pip install --upgrade --force --no-dependencies https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha2/tensorflow_macos-0.1a2-cp38-cp38-macosx_11_0_arm64.whl https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha2/tensorflow_addons_macos-0.1a2-cp38-cp38-macosx_11_0_arm64.whl
ERROR: tensorflow_macos-0.1a2-cp38-cp38-macosx_11_0_arm64.whl is not a supported wheel on this platform.

New conda environment, python 3.8.8.

Would you mind giving us a hand not sort this out?

mwidjaja1 commented 3 years ago

@epignatelli I'm sorry you're running into issues but based on what you sent me, I'm pretty sure you did not follow the Troubleshooting for importing TensorFlow instructions. I (who remember, is just a volunteer), explicitly asked you to provide very specific information when requesting for help to prove that you have followed the entire guide from start to finish.

Please reread the troubleshooting instructions, and once you read all the bullet points in that section, it should be pretty clear on what you need to include here for me to help out more.

Thanks!

Edit: Realized I must have backed it out on a previous edit so that's my fault [sighs at myself], but, please provide the terminal outputs you get while running the troubleshooting section.

lebigot commented 3 years ago

Thanks a lot!

The sanity check and troubleshooting sections are a useful idea. :)

Checking the Python interpreter path and architecture could be done more directly with:

> file $(which python)
…/miniforge3/bin/python: Mach-O 64-bit executable arm64
mwidjaja1 commented 3 years ago

@lebigot thank you for sharing that command! I knew there had to be an easier way but the internet was failing me. I'll update the instructions with your recommendation and give you credit. Many thanks!

lebigot commented 3 years ago

Thanks.

Steps 1 and 3 can be merged, too: with file, you get both the path and the type of executable. :)

pra-dan commented 3 years ago

Hey @mwidjaja1 Kudos for your efforts. Can you please share how your terminal looks that way ?

mwidjaja1 commented 3 years ago

...I really should have figured that out. Thanks @lebigot

@pra-dan To not flood this issue, I'd appreciate it if we can continue discussions outside of this issue if you have further questions. But to quickly rattle this off

edwin-yan commented 3 years ago

Thanks for putting together the instructions.I just noticed that 0.1a3 is out, so it would be great if you can update your instructions as well. I created the updated one for you:

X86: pip install --upgrade --force --no-dependencies https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_x86_64.whl https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_addons_macos-0.1a3-cp38-cp38-macosx_11_0_x86_64.whl ARM: pip install --upgrade --force --no-dependencies https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl https://github.com/apple/tensorflow_macos/releases/download/v0.1alpha3/tensorflow_addons_macos-0.1a3-cp38-cp38-macosx_11_0_arm64.whl