MacBook Pro M1: model will be on CPU

[eigenein]

It seems as if the --device were not passed down:

❯ python -m flux --name flux-dev --loop --device mps
Hardware accelerator e.g. GPU is available in the environment, but no `device` argument is passed to the `Pipeline` object. Model will be on CPU.

❯ git rev-parse HEAD
eae154e8a2a4aad791d52b34bcacb67ab6a9b8db

❯ pip freeze
altair==5.3.0
attrs==23.2.0
blinker==1.8.2
cachetools==5.4.0
certifi==2024.7.4
charset-normalizer==3.3.2
click==8.1.7
einops==0.8.0
filelock==3.15.4
fire==0.6.0
-e git+https://github.com/black-forest-labs/flux@eae154e8a2a4aad791d52b34bcacb67ab6a9b8db#egg=flux
fsspec==2024.6.1
gitdb==4.0.11
GitPython==3.1.43
huggingface-hub==0.24.5
idna==3.7
invisible-watermark==0.2.0
Jinja2==3.1.4
jsonschema==4.23.0
jsonschema-specifications==2023.12.1
markdown-it-py==3.0.0
MarkupSafe==2.1.5
mdurl==0.1.2
mpmath==1.3.0
networkx==3.3
numpy==2.0.1
opencv-python==4.10.0.84
packaging==24.1
pandas==2.2.2
pillow==10.4.0
protobuf==5.27.3
pyarrow==17.0.0
pydeck==0.9.1
Pygments==2.18.0
python-dateutil==2.9.0.post0
pytz==2024.1
PyWavelets==1.6.0
PyYAML==6.0.1
referencing==0.35.1
regex==2024.7.24
requests==2.32.3
rich==13.7.1
rpds-py==0.19.1
safetensors==0.4.3
sentencepiece==0.2.0
six==1.16.0
smmap==5.0.1
streamlit==1.37.0
streamlit-keyup==0.2.4
sympy==1.13.1
tenacity==8.5.0
termcolor==2.4.0
tokenizers==0.19.1
toml==0.10.2
toolz==0.12.1
torch==2.4.0
torchvision==0.19.0
tornado==6.4.1
tqdm==4.66.4
transformers==4.43.3
typing_extensions==4.12.2
tzdata==2024.1
urllib3==2.2.2

[bghira]

mps doesn't work at all. this model requires fp64

[Xuzzo]

@bghira There is a workaround. See #30

[mgierschdev]

the easiest way would be to convert it to a Apple Core ML model, similar to the other diffusion models, which will make possible to run it locally

[jardelva96]

Here are steps to troubleshoot and potentially resolve the issue:

Step-by-Step Resolution Ensure Correct --device Argument Passing: Ensure that the --device argument is being passed correctly to the Pipeline object within your script. You might need to modify the script to ensure it accepts and uses the --device argument appropriately.

Modify Your Script: If the script isn't handling the --device argument, you can modify it to include device handling. For example, in a PyTorch-based script, you would typically have something like: import torch def get_device(device_name): if device_name == 'mps': return torch.device('mps') if torch.backends.mps.is_available() else torch.device('cpu') elif device_name == 'cuda': return torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu') else: return torch.device('cpu')

Example usage: import argparse parser = argparse.ArgumentParser() parser.add_argument('--device', type=str, default='cpu', help='Device to run the model on') args = parser.parse_args() device = get_device(args.device) model.to(device) Check Compatibility: MPS currently supports only float32 tensors, and your model might be using float64 tensors. If this is the case, you can convert the model to float32. model = model.to(torch.float32) Convert to Core ML Model: If MPS support remains problematic, converting your model to an Apple Core ML model might be an effective workaround. This approach is similar to what is done with other diffusion models.

Use an Alternative Backend: If MPS doesn't support your model requirements (such as float64), you might need to use an alternative backend like CPU or CUDA (if you have a compatible NVIDIA GPU).

Example Script Modification: Here’s a snippet incorporating the --device argument correctly:

import argparse import torch

Function to get the appropriate device def get_device(device_name): if device_name == 'mps': return torch.device('mps') if torch.backends.mps.is_available() else torch.device('cpu') elif device_name == 'cuda': return torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu') else: return torch.device('cpu')

Argument parsing parser = argparse.ArgumentParser() parser.add_argument('--device', type=str, default='cpu', help='Device to run the model on') args = parser.parse_args() Select the device device = get_device(args.device) Assuming model is your model object model.to(device) print(f'Model is running on {device}') Workaround or Alternative Approaches Consult Issue #30: Refer to the workaround mentioned in issue #30 as suggested by @Xuzzo. This might provide additional insights or alternative solutions specific to your use case.

Apple Core ML Conversion: As @mgierschdev suggested, converting the model to Apple Core ML might be beneficial for running it on Apple Silicon efficiently.

[bghira]

nope. fp32 doesn't work

torch 2.4, m3 max

[jardelva96]

Script Modification Here’s a snippet incorporating the --device argument

import argparse import torch

def get_device(device_name): if device_name == 'mps': return torch.device('mps') if torch.backends.mps.is_available() else torch.device('cpu') elif device_name == 'cuda': return torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu') else: return torch.device('cpu')

parser = argparse.ArgumentParser() parser.add_argument('--device', type=str, default='cpu', help='Device to run the model on') args = parser.parse_args()

device = get_device(args.device) model = ... # Your model initialization model.to(device) print(f'Model is running on {device}')