search

domschl / HuggingFaceGuidedTourForMac

A guided tour on how to use HuggingFace large language models on Macs with Apple Silicon
MIT License
122 stars 14 forks source link
chatbot howto-tutorial huggingface llm nlp pytorch transformers

readme

Version

HuggingFace and Deep Learning guided tour for Macs with Apple Silicon

A guided tour on how to install optimized pytorch and optionally Apple's new MLX and/or Google's tensorflow or JAX on Apple Silicon Macs and how to use HuggingFace large language models for your own experiments. Recent Mac show good performance for machine learning tasks.

We will perform the following steps:

Then we provide additional HowTos for:

Additional overview notes Optional

(skip to 1. Preparations if you know which framework you are going to use)

What is Tensorflow vs. JAX vs. Pytorch vs. MLX and how relates Huggingface to it all?

Tensorflow, JAX, Pytorch, and MLX are deep-learning frameworks that provide the required libraries to perform optimized tensor operations used in training and inference. On high level, the functionality of all four is equivalent. Huggingface builds on top of any of the those frameworks and provides a large library of pretrained models for many different use-cases, ready to use or to customize plus a number of convenience libraries and sample code for easy getting-started.

HuggingFace publishes an Overview of model-support for each framework. Currently, Pytorch is the defacto standard, if you want to make use of existing models.

Note: For the (probably too simplified) answer to the question "What's the fastest?" have a look at the Jupyter notebook 02-Benchmarks, and once you've completed the installation, you can test your own environment. The notebook allows to compare the speed of matrix multiplications for different frameworks. However, the difference between frameworks when performing 'standard' model training or inference tasks will most likely be less pronounced.

1. Preparations

1.1 Install homebrew

If you haven't done so, go to https://brew.sh/ and follow the instructions to install homebrew. Once done, open a terminal and type brew --version to check that it is installed correctly.

Now use brew to install more recent versions of python and git. The recommendation is to use Homebrew's default Python 3.12, if you are not planning to use Tensorflow with Metal optimization (still requires 3.11 (at 2024-04)).

Current Python for Huggingface, Pytorch, JAX, and MLX, Python 3.12, Homebrew default

brew install python@3.12 git

Legacy installations (Tensorflow), Python 3.11 Optional

brew install python@3.11 git

Note: you can install both versions of Python and then create a virtual environment using the specific python version you need for each case.

Note: If you plan to also use Linux, be aware that Python version support sometimes differs between Mac and Linux version of frameworks.

Make homebrew's Python the system-default Optional

Note: Apple does not put too much energy into keeping MacOS's python up-to-date. If you want to use an up-to-date default python, it makes sense to make homebrew's python the default system python. So, if, you want to use homebrew's Python 3.11 or 3.12 system-globally, the easiest way way to do so (after brew install python@3.12 or 3.11):

Edit ~/.zshrc and insert:

# This is OPTIONAL and only required if you want to make homebrew's Python 3.12 as the global version:
export PATH="/opt/homebrew/opt/python@3.12/bin:$PATH"                     
export PATH=/opt/homebrew/opt/python@3.12/libexec/bin:$PATH

Change all references of 3.12 to 3.11 when wanting to make homebrew's Python 3.11 system-standard python.

(Restart your terminal to activate the path changes, or enter source ~/.zshrc in your current terminal session.)

Note: Regardless of the system python in use, when creating a virtual environment, you can always select the specific python version you want to use in the venv by creating the venv with exactly that python. E.g. /usr/bin/python3 -m venv my_venv_name creates a virtual environment using Apple's macOS python (which at the time of this writing, 2024-07, is still stuck at 3.9.6). See below, Virtual environments, for more details.

1.2 Test project

Now clone this project as a test project:

git clone https://github.com/domschl/HuggingFaceGuidedTourForMac

This clones the test-project into a directory HuggingFaceGuidedTourForMac

Virtual environment

Now create a Python 3.12 environment for this project and activate it:

(Again: replace with 3.11, if you need)

python3.12 -m venv HuggingFaceGuidedTourForMac

Creating a venv adds the files required (python binaries, libraries, configs) for the virtual python environment to the project folder we just cloned, using again the same directory HuggingFaceGuidedTourForMac. Enter the directory and activate the virtual environment:

cd HuggingFaceGuidedTourForMac
source bin/activate

Now the directory HuggingFaceGuidedTourForMac contains the content of the github repository (e.g. 00-SystemCheck.ipynb) and the the files for the virtual env (e.g. bin, lib, etc, include, share, pyvenv.cfg):

Folder content

Alternatives: If you have many different python versions installed, you can create an environment that uses a specific version by specifying the path of the python that is used to create the venv, e.g.: 

/opt/homebrew/opt/python@3.12/bin/python3.12 -m venv my_new_312_env

uses homebrew's python explicitly to create a new venv, whereas

/usr/bin/python3 -m venv my_old_system_venv

would use Apple's macOS python version for the new environment.

1.3 When you done with your project

Do deactivate this virtual environment, simply use:

deactivate

To re-activate it, enter the directory that contains the venv, here: HuggingFaceGuidedTourForMac and use:

source bin/activate

Additional notes on venv Optional

Warning A very unintuitive property of venv is the fact: while you enter an environment by activating it in the subdirectory of your project (with source bin/activate), the venv stays active when you leave the project folder and start working on something completely different until you explicitly deactivate the venv with deactivate.

There are a number of tools that modify the terminal system prompt to display the currently active venv, which is very helpful thing. Check out starship (recommended), or, if you like embellishment Oh My Zsh.

No venv active Example with powerlevel10k installed. The left side of the system prompt shows the current directory, the right side would show the name of the venv. Currently, no venv is active.

After activating a venv in HuggingFaceGuidedTourForMac:

venv is still active Even is the working directoy is changed (here to home), since the venv is still active, it's name is displayed on the right side by powerlevel10k. Very handy.

Note: See https://docs.python.org/3/tutorial/venv.html for more information about Python virtual environments.

2 Install pytorch

Make sure that your virtual environment is active with pip -V (uppercase V), this should show a path for pip within your project:

<your-path>/HuggingFaceGuidedTourForMac/lib/python3.12/site-packages/pip (python 3.12)

Following https://pytorch.org, we will install Pytorch with pip. You need at least version 2.x (default since 2023) in order to get MPS (Metal Performance Shaders) support within pytorch, which offers significant performance advantage on Apple Silicon.

To install pytorch into the venv:

pip install -U torch numpy torchvision torchaudio

2.1 Quick-test pytorch

To test that pytorch is installed correctly, and MPS metal performance shaders are available, open a terminal, type python and within the python shell, enter:

import torch
# check if MPS is available:
torch.backends.mps.is_available()

This should return True.

3 Install MLX Optional

pip install -U mlx

3.1 Quick-test MLX

Again, start python and enter:

import mlx.core as mx
print(mx.__version__)

This should print a version, such as 0.16.1 (2024-07)

4.1 Install JAX Optional

JAX is an excellent choice, if low-level optimization of algorithms and research beyond the boundaries of established deep-learning algorithms is your focus. Modelled after numpy, it supports automatic differentiation of 'everything' (for optimization problems) and supports vectorization and parallelization of python algorithms beyond mere deep learning. To get functionality that is expected from other deep learning frameworks (layers, training-loop functions and similar 'high-level'), consider installing additional neural network library such as: flax.

Check supported versions

Unfortunately, the JAX metal drivers have started to lag behind JAX releases, and therefore you need to check the compatibility table for the supported versions of JAX that match the available jax-metal drivers.

To install a specific version of JAX and the latest jax-metal with pip into the active environment:

# The version 0.4.26 is taken from the compatibility table mentioned above. Update as required.
pip install -U jax==0.4.26 jaxlib==0.4.26 jax-metal

4.2 Quick-test JAX

Start python (3.12 is supported) and enter:

import jax
print(jax.devices()[0])

This should display (on first run only):

Platform 'METAL' is experimental and not all JAX functionality may be correctly supported!
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
W0000 00:00:1721975334.430133   43061 mps_client.cc:510] WARNING: JAX Apple GPU support is experimental and not all JAX functionality is correctly supported!
Metal device set to: Apple M2 Max

systemMemory: 32.00 GB
maxCacheSize: 10.67 GB

I0000 00:00:1721975334.446739   43061 service.cc:145] XLA service 0x60000031d100 initialized for platform METAL (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1721975334.446771   43061 service.cc:153]   StreamExecutor device (0): Metal, <undefined>
I0000 00:00:1721975334.448269   43061 mps_client.cc:406] Using Simple allocator.
I0000 00:00:1721975334.448308   43061 mps_client.cc:384] XLA backend will use up to 22906109952 bytes on device 0 for SimpleAllocator.
[METAL(id=0)]

Here METAL:0 is the device that JAX will use for calculations, and Apple Silicon is supported.

Errors

If, instead you see errors like:

RuntimeError: Unable to initialize backend 'METAL': INVALID_ARGUMENT: Mismatched PJRT plugin PJRT API version (0.47) and framework PJRT API version 0.54). (you may need to uninstall the failing plugin package, or set JAX_PLATFORMS=cpu to skip this backend.)

Your version of jax and jaxlib are incompatible with jax-metal. Check the compatibility table for jax-metal and install the required versions as indicated in the table.

4.3 Install tensorflow Optional

Warning: Tensorflow is losing support fast, and not even Google publishes new models for Tensorflow. A migration plan is recommended, if you plan to use this.

Note: While Tensorflow supports Python 3.12 since 2.16, the macOS tensorflow-metal accelerator has not been updated since 2023-09 (status of 2024-07) and requires Python 3.11:

Make sure that your virtual environment is active with pip -V (uppercase V), this should show a path for pip within your project:

<your-path>/HuggingFaceGuidedTourForMac/lib/python3.11/site-packages/pip (python 3.11)

Following https://developer.apple.com/metal/tensorflow-plugin/, we will install tensorflow with pip within our venv:

pip install -U tensorflow tensorflow-metal

4.4 Quick-test Tensorflow

To test that tensorflow is installed correctly, open a terminal, type python and within the python shell, enter:

import tensorflow as tf
tf.config.list_physical_devices('GPU')

You should see something like:

[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

5 Jupyter lab

At this point, your Apple Silicon Mac should be ready to run pytorch and optionally MLX and/or JAX or tensorflow with hardware acceleration support, using the Apple Metal framework.

To test this, you can use jupyter lab to run some notebooks. To install jupyter lab, first make sure the virtual environment you want to use is active (pip -V), and type:

pip install -U jupyterlab ipywidgets

Note: If you have other Jupyter versions installed, the path to the newly installed jupyter version within the venv is often not updated correctly, re-activate the environment to make sure that the correct local Jupyter version is used:

deactivate
source bin/activate

To start Jupyter lab, type:

jupyter lab

This should open a browser window with jupyter lab running. You can then create a new python notebook and run some code to test that tensorflow and pytorch are working correctly:

import torch

print("Pytorch version:", torch.__version__)

If this completed successful, your Mac is now ready for Deep Learning experiments.

6 HuggingFace

HuggingFace is a great resource for NLP and Deep Learning experiments. It provides a large number of pre-trained language models and a simple API to use them. It will allow us to quickly get started with Deep Learning experiments.

6.1 Install transformers

From the huggingface installation instructions, we use pip to install transformers:

pip install -U transformers accelerate "huggingface_hub[cli]"

Note: When experimenting with HuggingFace, you will download large models that will be stored in your home directory at: ~/.cache/huggingface/hub. You can remove these models at any time by deleting this directory or parts of it's content.

7 Experiments

7.1 Simple sentiment analysis

Within the directory HuggingFaceGuidedTourForMac and active venv, start jupyter lab and load the 00-SystemCheck.ipynb notebook. The notebook will first check all the deep-learning frameworks and give information, if they are correctly installed. Afterward, Pytorch is used for a simple experiment.

Use <Shift>-Enter to run the notebook's cells.

Note: If you started Jupyter Lab before installing Huggingface, you either need to restart the python kernel in Jupyter or simply restart Jupyter Lab, otherwise it won't find the Transformers library.

After the various tests, your should finally see something like this:

If you've received a label classification of POSITIVE with a score of 0.99, then you are ready to start experimenting with HuggingFace!

Note: You'll see that the HuggingFace libraries are downloading all sorts of large binary blobs containing the trained model data. That data is stored in your home directory at: ~/.cache/huggingface/hub. You can remove these models at any time by deleting this directory or parts of it's content.

Trouble-shooting

7.2 Minimal chat-bot

You can open the notebook 01-ChatBot.ipynb to try out a very simple chatbot on your Mac.

The python code used is:

import torch 
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.utils import logging

# Disable warnings about padding_side that cannot be rectified with current software:
logging.set_verbosity_error()

model_names = ["microsoft/DialoGPT-small", "microsoft/DialoGPT-medium", "microsoft/DialoGPT-large"]
use_model_index = 1  # Change 0: small model, 1: medium, 2: large model (requires most resources!)
model_name = model_names[use_model_index]

tokenizer = AutoTokenizer.from_pretrained(model_name) # , padding_side='left')
model = AutoModelForCausalLM.from_pretrained(model_name)

# The chat function: received a user input and chat-history and returns the model's reply and chat-history:
def reply(input_text, history=None):
    # encode the new user input, add the eos_token and return a tensor in Pytorch
    new_user_input_ids = tokenizer.encode(input_text + tokenizer.eos_token, return_tensors='pt')

    # append the new user input tokens to the chat history
    bot_input_ids = torch.cat([history, new_user_input_ids], dim=-1) if history is not None else new_user_input_ids

    # generated a response while limiting the total chat history to 1000 tokens, 
    chat_history_ids = model.generate(bot_input_ids, max_length=1000, pad_token_id=tokenizer.eos_token_id)

    # pretty print last ouput tokens from bot
    return tokenizer.decode(chat_history_ids[:, bot_input_ids.shape[-1]:][0], skip_special_tokens=True), chat_history_ids

history = None
while True:
    input_text = input("> ")
    if input_text in ["", "bye", "quit", "exit"]:
        break
    reply_text, history_new = reply(input_text, history)
    history=history_new
    if history.shape[1]>80:
        old_shape = history.shape
        history = history[:,-80:]
        print(f"History cut from {old_shape} to {history.shape}")
    # history_text = tokenizer.decode(history[0])
    # print(f"Current history: {history_text}")
    print(f"D_GPT: {reply_text}")

This shows a (quite limited and repetitive) chatbot using Microsoft's DialoGPT models.

Things to try:

Next steps

Learning resources

Conda uninstallation notes

Note: This paragraph is to uninstall conda that was used in older versions of this guide:

brew uninstall miniconda

Additional modifications are (all of them are inactive, once miniconda is removed):

Changes