Convert your ControlNet model to A1111/lllyasviel format

[continue-revolution]

I am the author of sd-webui-segment-anything. The only thing your work is not compatible with Mikubill ControlNet extension and my SAM extension is your ControlNet model format. Please convert your ControlNet model to lllyasviel format, similar to this (1 yaml + 1 model state dict, the state dict keys should be the same as other models here, the yaml filename should be the same as the model filename, except the file extension) and your work can soon be compatible with A1111 sd-webui and be accessible to the broad stable diffusion community.

[gasvn]

The model is converted to support diffusers package. Do you mean that you need the unconverted model?

[continue-revolution]

Yes

[gasvn]

I have an old ckpt in hugglingface: https://huggingface.co/shgao/edit-anything-v0-1/tree/main And the config file is https://github.com/sail-sg/EditAnything/blob/main/models/cldm_v21.yaml Can you please check if these two files are want you need. If so, I'll update the hugglingface and upload the new model with this format. Thanks~

[continue-revolution]

I have checked. Just need one more step which is something like this. I do not need base model. Thanks for your help.

BTW, do you think your model can work for SD1.5? The majority of community models are based on sd1.5, so if your models are compatible with SD1.5, it would be very appealing to the SD community (including a large amount of non-research, or even non-CS people).

import torch
sd = torch.load('../tmp/edit-anything-ckpt-v0-1.ckpt')['state_dict']
idx = 0
sd_cnet_only = {}
for key, val in sd.items():
    if 'control_model' in key:
        sd_cnet_only[key] = val
torch.save(sd_cnet_only, '../tmp/control_v11p_sd15_sam.pth')

[gasvn]

Thanks, I will upload a new model with the required form. And I can train a SD1.5 model for the community. I will keep you updated once I finished the training.

[gasvn]

@continue-revolution hi, I have trained new sam condition models based on sd15 and sd21. These models are trained with more images and longer iterations. I have converted the format as you requested https://huggingface.co/shgao/edit-anything-v0-4-lllyasviel-format/tree/main

[continue-revolution]

Thank you very much, this is very helpful. I will test and update a tutorial for people who use A1111 SD webui. Appreciate your effort and contribution.

[alelordelo]

@continue-revolution , did you port this to A1111 SD webui? : )

[continue-revolution]

Yes, but there seem to be some problem since I didn’t get the expected performance. I suspect that my code for detect map (control image) might be problematic and I have to try their demo to see what their detection map looks like, but recently I fell into an extremely busy life so that I have not tried their demo yet. I cannot give you an exact time on when it could me available, but if @gasvn can show me a detect map for a sample image then the progress could be significantly speed up.

[gasvn]

@continue-revolution Can you share me with the code link so I can check it?

[continue-revolution]

https://github.com/continue-revolution/sd-webui-segment-anything/blob/master/scripts/auto.py#L73-L93

Another probably useful way is to show me a detect map (not random segmentation, I know that is not detect map) for whatever image

[gasvn]

Sorry for the late reply. You can copy the function in https://github.com/sail-sg/EditAnything/blob/10374ef2a9fc92e272691c38d8cbd39f15ddf45c/sam2edit_lora.py#L354 to replace your sam condition generation.

[continue-revolution]

https://github.com/sail-sg/EditAnything/blob/main/sam2image.py#L95

It seems that this line is unnecessary. Why did you ever sort it?

[continue-revolution]

I was finally able to create some meaningful image. I will soon be publishing this support to my extension

Edit: published

[alelordelo]

@continue-revolution , the plugin is for SAM generated map -> SD -> output? Can you prompt per mask region?

[continue-revolution]

@alelordelo you can do this one time per each mask region. I will look into the controlnet source code to see if it is possible to generate one image with different condition/uncondition for different mask regions.

[continue-revolution]

@alelordelo Today I looked into the source code of A1111 and I would say yes, it IS possible to do this. I think your proposal is really cool and will be supported in near furture.

[alelordelo]

awesome! I am working on a dataset for this. DO you think COCO JSON would be a good fit?

[alelordelo]

@continue-revolution