Benchmark against PyTorch with Accelerate enabled

certik commented 1 year ago

So far we only benchmarked against PyTorch+OpenBLAS. We should also benchmark against PyTorch+Accelerate.

Here are a few ways how to do it:

https://towardsdatascience.com/installing-pytorch-on-apple-m1-chip-with-gpu-acceleration-3351dc44d67c
https://twitter.com/isuru_f/status/1636013111592329216 (conda install "libblas=*=*_accelerate")

certik commented 1 year ago

I did:

$ mamba create -n pytorch2 transformers "libblas=*=*_accelerate"
$ conda activate pytorch2
$ python pt.py
Importing
Traceback (most recent call last):
  File "/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1126, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/transformers/pipelines/__init__.py", line 45, in <module>
    from .audio_classification import AudioClassificationPipeline
  File "/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/transformers/pipelines/audio_classification.py", line 21, in <module>
    from .base import PIPELINE_INIT_ARGS, Pipeline
  File "/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/transformers/pipelines/base.py", line 36, in <module>
    from ..modelcard import ModelCard
  File "/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/transformers/modelcard.py", line 48, in <module>
    from .training_args import ParallelMode
  File "/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/transformers/training_args.py", line 29, in <module>
    from .debug_utils import DebugOption
  File "/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/transformers/debug_utils.py", line 21, in <module>
    import torch
  File "/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/__init__.py", line 218, in <module>
    from torch._C import *  # noqa: F403
    ^^^^^^^^^^^^^^^^^^^^^^
ImportError: dlopen(/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/_C.cpython-311-darwin.so, 0x0002): Library not loaded: @rpath/libopenblas.0.dylib
  Referenced from: /Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/libtorch_cpu.dylib
  Reason: tried: '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/../../../../libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/../../../../libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/../../../../libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/../../../../libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/../../../libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/../../../libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/bin/../lib/libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/bin/../lib/libopenblas.0.dylib' (no such file), '/usr/local/lib/libopenblas.0.dylib' (no such file), '/usr/lib/libopenblas.0.dylib' (no such file)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/ondrej/repos/fastGPT/pt.py", line 5, in <module>
    from transformers import pipeline
  File "<frozen importlib._bootstrap>", line 1231, in _handle_fromlist
  File "/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1116, in __getattr__
    module = self._get_module(self._class_to_module[name])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1128, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.pipelines because of the following error (look up to see its traceback):
dlopen(/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/_C.cpython-311-darwin.so, 0x0002): Library not loaded: @rpath/libopenblas.0.dylib
  Referenced from: /Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/libtorch_cpu.dylib
  Reason: tried: '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/../../../../libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/../../../../libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/../../../../libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/../../../../libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/../../../libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/lib/libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/torch/../../../libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/bin/../lib/libopenblas.0.dylib' (no such file), '/Users/ondrej/mambaforge/envs/pytorch2/bin/../lib/libopenblas.0.dylib' (no such file), '/usr/local/lib/libopenblas.0.dylib' (no such file), '/usr/lib/libopenblas.0.dylib' (no such file)

I then tried:

$ mamba install openblas
$ python pt.py
Importing
  Time:  0.47915499983355403
Loading
  Time:  4.010167125146836
Generating
/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/transformers/generation/utils.py:1201: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation)
  warnings.warn(
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
  Time:  0.8942654579877853
Alan Turing theorized that computers would one day become very powerful, but even he could not imagine how they would be able to do so.

"I think that the most important thing is

I tried modifying pt.py to run on 2, 4, 8 cores. It runs with the same times as with OpenBLAS, so I am assuming it is using OpenBLAS.

If anyone knows how to run PyTorch using Accelerate, let me know.

isuruf commented 1 year ago

This is a packaging bug. Try

ln -sf $CONDA_PREFIX/lib/libcblas.3.dylib $CONDA_PREFIX/lib/libopenblas.0.dylib

certik commented 1 year ago

@isuruf awesome, thanks! Now it works:

$ mamba create -n pytorch2 transformers "libblas=*=*_accelerate"
$ conda activate pytorch2
$ ln -sf $CONDA_PREFIX/lib/libcblas.3.dylib $CONDA_PREFIX/lib/libopenblas.0.dylib
$ python pt.py
Importing
  Time:  0.46954312501475215
Loading
  Time:  2.912111165933311
Generating
/Users/ondrej/mambaforge/envs/pytorch2/lib/python3.11/site-packages/transformers/generation/utils.py:1201: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation)
  warnings.warn(
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
  Time:  0.34642862528562546
Alan Turing theorized that computers would one day become very powerful, but even he could not imagine how they would be able to do so.

"I think that the most important thing is

The timing is 0.346s. fastGPT with Accelerate gets 0.302s.

certik / fastGPT

Benchmark against PyTorch with Accelerate enabled #26