regisss
commented
3 weeks ago I cannot reproduce it. How did you install optimum-habana and what was the last commit?
Closed KiwiHana closed 2 weeks ago
regisss
commented
3 weeks ago I cannot reproduce it. How did you install optimum-habana and what was the last commit?
KiwiHana
commented
2 weeks ago I cannot reproduce it. How did you install optimum-habana and what was the last commit?
My device is Gaudi2D.
I installed on August 23th.
pip install git+https://github.com/huggingface/optimum-habana.git
git clone https://github.com/huggingface/optimum-habanaIt is OK for SDv2.1 FP32 precision like:
python text_to_image_generation.py --model_name_or_path /data/sd/stable-diffusion-2-1-base --prompts "An image of a squirrel in Picasso style" --num_images_per_prompt 28 --batch_size 1 --height 512 --width 512 --image_save_dir /tmp/stable_diffusion_images --use_habana --use_hpu_graphs --gaudi_config Habana/stable-diffusion-2
But error for SDv2.1 BF16 precision. RuntimeError: synNodeCreateWithId failed for node: spatial_convolution with synStatus 26 [Generic failure].
python text_to_image_generation.py --model_name_or_path /data/sd/stable-diffusion-2-1-base --prompts "An image of a squirrel in Picasso style" --num_images_per_prompt 28 --batch_size 1 --height 512 --width 512 --image_save_dir /tmp/stable_diffusion_images --use_habana --use_hpu_graphs --gaudi_config Habana/stable-diffusion-2 --bf16
KiwiHana
commented
2 weeks ago python text_to_image_generation.py --model_name_or_path /data/stable-diffusion-2-1-base --prompts "An image of a squirrel in Picasso style" --num_images_per_prompt 28 --batch_size 1 --height 1920 --width 1080 --image_save_dir /data/stable_diffusion_images --use_habana --use_hpu_graphs --gaudi_config /data/stable-diffusion --bf16
/usr/local/lib/python3.10/dist-packages/diffusers/models/vq_model.py:20: FutureWarning: `VQEncoderOutput` is deprecated and will be removed in version 0.31. Importing `VQEncoderOutput` from `diffusers.models.vq_model` is deprecated and this will be removed in a future version. Please use `from diffusers.models.autoencoders.vq_model import VQEncoderOutput`, instead.
deprecate("VQEncoderOutput", "0.31", deprecation_message)
/usr/local/lib/python3.10/dist-packages/diffusers/models/vq_model.py:25: FutureWarning: `VQModel` is deprecated and will be removed in version 0.31. Importing `VQModel` from `diffusers.models.vq_model` is deprecated and this will be removed in a future version. Please use `from diffusers.models.autoencoders.vq_model import VQModel`, instead.
deprecate("VQModel", "0.31", deprecation_message)
Loading pipeline components...: 100%|██████████████████████████████████████████████| 6/6 [00:00<00:00, 9.28it/s]
[INFO|pipeline_utils.py:130] 2024-08-27 04:45:04,526 >> Enabled HPU graphs.
/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py:366: UserWarning: torch.distributed.reduce_op is deprecated, please use torch.distributed.ReduceOp instead
warnings.warn(
[INFO|configuration_utils.py:303] 2024-08-27 04:45:04,656 >> loading configuration file /data/stable-diffusion/gaudi_config.json
[INFO|configuration_utils.py:358] 2024-08-27 04:45:04,656 >> GaudiConfig {
"autocast_bf16_ops": null,
"autocast_fp32_ops": null,
"optimum_version": "1.12.0",
"transformers_version": "4.43.3",
"use_dynamic_shapes": false,
"use_fused_adam": true,
"use_fused_clip_norm": true,
"use_torch_autocast": true
}
[WARNING|pipeline_utils.py:156] 2024-08-27 04:45:04,656 >> `use_torch_autocast` is True in the given Gaudi configuration but `torch_dtype=torch.bfloat16` was given. Disabling mixed precision and continuing in bf16 only.
============================= HABANA PT BRIDGE CONFIGURATION ===========================
PT_HPU_LAZY_MODE = 1
PT_RECIPE_CACHE_PATH =
PT_CACHE_FOLDER_DELETE = 0
PT_HPU_RECIPE_CACHE_CONFIG =
PT_HPU_MAX_COMPOUND_OP_SIZE = 9223372036854775807
PT_HPU_LAZY_ACC_PAR_MODE = 1
PT_HPU_ENABLE_REFINE_DYNAMIC_SHAPES = 0
---------------------------: System Configuration :---------------------------
Num CPU Cores : 192
CPU RAM : 2113382084 KB
------------------------------------------------------------------------------
[INFO|pipeline_stable_diffusion.py:411] 2024-08-27 04:45:09,023 >> 1 prompt(s) received, 28 generation(s) per prompt, 1 sample(s) per batch, 28 total batch(es).
0%| | 0/28 [00:00<?, ?it/s]/usr/local/lib/python3.10/dist-packages/diffusers/models/unets/unet_2d_blocks.py:1369: FutureWarning: `scale` is deprecated and will be removed in version 1.0.0. The `scale` argument is deprecated and will be ignored. Please remove it, as passing it will raise an error in the future. `scale` should directly be passed while calling the underlying pipeline component i.e., via `cross_attention_kwargs`.
deprecate("scale", "1.0.0", deprecation_message)
/usr/local/lib/python3.10/dist-packages/diffusers/models/unets/unet_2d_blocks.py:2628: FutureWarning: `scale` is deprecated and will be removed in version 1.0.0. The `scale` argument is deprecated and will be ignored. Please remove it, as passing it will raise an error in the future. `scale` should directly be passed while calling the underlying pipeline component i.e., via `cross_attention_kwargs`.
deprecate("scale", "1.0.0", deprecation_message)
57%|███████████████████████████████████████████▍ | 16/28 [22:37<15:14, 76.25s/it]
System Info
Information
Tasks
examplesfolder (such as GLUE/SQuAD, ...)Reproduction
cd /data/optimum-habana/examples/stable-diffusion
python text_to_image_generation.py --model_name_or_path /data/sd/stable-diffusion-2-1-base --prompts "An image of a squirrel in Picasso style" --num_images_per_prompt 28 --batch_size 1 --height 512 --width 512 --image_save_dir /tmp/stable_diffusion_images --use_habana --use_hpu_graphs --gaudi_config Habana/stable-diffusion-2 --bf16
Expected behavior