patrickvonplaten
commented
1 year ago Hey @depyronick,
Could you copy-paste a reproducible code snippet?
Open depyronick opened 1 year ago
patrickvonplaten
commented
1 year ago Hey @depyronick,
Could you copy-paste a reproducible code snippet?
depyronick
commented
1 year ago def base64_to_pil(base64_image_data: str):
return Image.open(BytesIO(base64.b64decode(base64_image_data)))
image = base64_to_pil('...base64stringhere')
pipeline = StableDiffusionControlNetPipeline.from_pretrained(
MODEL_ID,
torch_dtype=torch.float16,
local_files_only=True,
controlnet=ControlNetModel.from_pretrained(
"./controlnets/openpose", torch_dtype=torch.float16),
safety_checker=None
).to('cuda')
pipeline.scheduler = UniPCMultistepScheduler.from_config(
pipeline.scheduler.config)
pipeline.enable_xformers_memory_efficient_attention()
pipeline.enable_model_cpu_offload()
body_model = Body('./controlnets/net/annotator/ckpts/body_pose_model.pth')
openpose = OpenposeDetector(body_model)
image = openpose(image)
pipeline(
prompt=prompt,
height=height,
width=width,
generator=generator,
negative_prompt=negative_prompt,
num_images_per_prompt=num_images_per_prompt,
num_inference_steps=num_inference_steps,
guidance_scale=guidance_scale,
image=image
)Hey @depyronick,
Could you copy-paste a reproducible code snippet?
jinwonkim93
commented
1 year ago Have you check your image? @depyronick
depyronick
commented
1 year ago Have you check your image? @depyronick
what do you mean by that?
jinwonkim93
commented
1 year ago Have you check your image? @depyronick
what do you mean by that?
image = base64_to_pil('...base64stringhere')
can you check if this is valid image?
depyronick
commented
1 year ago Have you check your image? @depyronick
what do you mean by that?
image = base64_to_pil('...base64stringhere')
can you check if this is valid image?
Yes.
and base64_to_pil returns: <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=480x600 at 0x7F309E68BFA0>. It also produces the same image if saved to disk.
same image works with other nets, just no with openpose.
jinwonkim93
commented
1 year ago Have you check your image? @depyronick
what do you mean by that?
image = base64_to_pil('...base64stringhere') can you check if this is valid image?
Yes.
Base 64 Image and
base64_to_pilreturns:<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=480x600 at 0x7F309E68BFA0>. It also produces the same image if saved to disk.same
imageworks with other nets, just no with openpose.
well i tried your code. i am not sure if you use same weight with lllyasviel, it worked.
depyronick
commented
1 year ago Have you check your image? @depyronick
what do you mean by that?
image = base64_to_pil('...base64stringhere') can you check if this is valid image?
Yes. Base 64 Image and
base64_to_pilreturns:<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=480x600 at 0x7F309E68BFA0>. It also produces the same image if saved to disk. sameimageworks with other nets, just no with openpose.well i tried your code. i am not sure if you use same weight with lllyasviel, it worked.
Interesting.
I am using annotators from
https://huggingface.co/lllyasviel/ControlNet
and ControlNetModel from
https://huggingface.co/fusing/stable-diffusion-v1-5-controlnet-openpose controlnets/openpose
can you pleas share the code that you were able to make it work?
jinwonkim93
commented
1 year ago from diffusers import StableDiffusionControlNetPipeline, ControlNetModel, UniPCMultistepScheduler
from controlnet_aux import OpenposeDetector
from controlnet_aux.open_pose import Body
import torch
def base64_to_pil(base64_image_data: str):
return Image.open(BytesIO(base64.b64decode(base64_image_data)))
image = base64_to_pil('...base64stringhere')
pipeline = StableDiffusionControlNetPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
torch_dtype=torch.float16,
local_files_only=True,
controlnet=ControlNetModel.from_pretrained(
"lllyasviel/sd-controlnet-openpose", torch_dtype=torch.float16),
safety_checker=None
).to('cuda')
pipeline.scheduler = UniPCMultistepScheduler.from_config(
pipeline.scheduler.config)
pipeline.enable_xformers_memory_efficient_attention()
pipeline.enable_model_cpu_offload()
body_model = Body('./models--lllyasviel--ControlNet/snapshots/e78a8c4a5052a238198043ee5c0cb44e22abb9f7/annotator/ckpts/body_pose_model.pth')
openpose = OpenposeDetector(body_model)
image = openpose(image)
output = pipeline(
prompt="gundam",
height=512,
width=512,
image=image
)
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel, UniPCMultistepScheduler from controlnet_aux import OpenposeDetector from controlnet_aux.open_pose import Body import torch def base64_to_pil(base64_image_data: str): return Image.open(BytesIO(base64.b64decode(base64_image_data))) image = base64_to_pil('...base64stringhere') pipeline = StableDiffusionControlNetPipeline.from_pretrained( "runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16, local_files_only=True, controlnet=ControlNetModel.from_pretrained( "lllyasviel/sd-controlnet-openpose", torch_dtype=torch.float16), safety_checker=None ).to('cuda') pipeline.scheduler = UniPCMultistepScheduler.from_config( pipeline.scheduler.config) pipeline.enable_xformers_memory_efficient_attention() pipeline.enable_model_cpu_offload() body_model = Body('./models--lllyasviel--ControlNet/snapshots/e78a8c4a5052a238198043ee5c0cb44e22abb9f7/annotator/ckpts/body_pose_model.pth') openpose = OpenposeDetector(body_model) image = openpose(image) output = pipeline( prompt="gundam", height=512, width=512, image=image )
what's your cv2 version?
jinwonkim93
commented
1 year ago from diffusers import StableDiffusionControlNetPipeline, ControlNetModel, UniPCMultistepScheduler from controlnet_aux import OpenposeDetector from controlnet_aux.open_pose import Body import torch def base64_to_pil(base64_image_data: str): return Image.open(BytesIO(base64.b64decode(base64_image_data))) image = base64_to_pil('...base64stringhere') pipeline = StableDiffusionControlNetPipeline.from_pretrained( "runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16, local_files_only=True, controlnet=ControlNetModel.from_pretrained( "lllyasviel/sd-controlnet-openpose", torch_dtype=torch.float16), safety_checker=None ).to('cuda') pipeline.scheduler = UniPCMultistepScheduler.from_config( pipeline.scheduler.config) pipeline.enable_xformers_memory_efficient_attention() pipeline.enable_model_cpu_offload() body_model = Body('./models--lllyasviel--ControlNet/snapshots/e78a8c4a5052a238198043ee5c0cb44e22abb9f7/annotator/ckpts/body_pose_model.pth') openpose = OpenposeDetector(body_model) image = openpose(image) output = pipeline( prompt="gundam", height=512, width=512, image=image )what's your cv2 version?
opencv-python-headless==4.5.5.64
No matter what I've tried, openpose not working. Other control nets working but not pose.