search

aws-neuron / aws-neuron-samples

Example code for AWS Neuron SDK developers building inference and training applications
Other
101 stars 32 forks source link

Can't get ControlNet on Inf2 to work #25

Open FraPochetti opened 10 months ago

FraPochetti commented 10 months ago

Hi team, I am adapting this notebook, essentially instantiating a ControlNet pipe, such as

controlnet = ControlNetModel.from_pretrained("DionTimmer/controlnet_qrcode-control_v1p_sd15",
                                             torch_dtype=torch.float16)

pipe = StableDiffusionControlNetImg2ImgPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    controlnet=controlnet,
    safety_checker=None,
    torch_dtype=torch.float16
)

and then going for torch_neuronx.trace I suspect the number 1 blocker is the fact that the main libs the notebooks suggest to install

!pip install diffusers==0.14.0 transformers==4.30.2 accelerate==0.16.0 safetensors==0.3.1 matplotlib

are too "old" for ControlNet. For instance, transformers and accelerate need to be upgraded. This prompts the update of other dependencies and then I end up with an env that is completely different that the originally recommended one. This causes, for instance, torch complaining about CUDA (among other things), whereas we are on Inf2 (this is confusing). Tried multiple times but somehow couldn't get very far. I also tried with the latest Neuron release and I can't get it to work.

Any help would be massively appreciated!

hannanjgaws commented 10 months ago

Hi @FraPochetti:

Thank you for raising this. Enabling ControlNet is on our roadmap and support will be enabled in a future Neuron release. We'll keep this ticket open until support is available.