search

modelscope / scepter

SCEPTER is an open-source framework used for training, fine-tuning, and inference with generative models.
https://github.com/modelscope/scepter
Apache License 2.0
428 stars 23 forks source link

segmentation-based controlled image generation #51

Open twinkleyang1 opened 1 month ago

twinkleyang1 commented 1 month ago

Thank you for your contributions. Could you please tell me how to do segmentation-based controlled image generation

jiangzeyinzi commented 1 month ago

Please check this #8

twinkleyang1 commented 1 month ago

@jiangzeyinzi Thank you for your answer. I made the modification on the yaml file based on canny, but now I have encountered some problems. My data collection format Source: FILE, Target: FILE, Prompt. Source:FILE indicates the original image; Target:FILE indicates the segmentation image; Prompt: Convert to a segmentation map based on the prompt. But the result is similar to Target:FILE. I want to generate images that are similar to Source:FILE and based on the conditions of Target:FILE

jiangzeyinzi commented 1 month ago

Based on your description, I'm still a bit confused about which task you would like to execute: 1) Conditional generation task: Generate an image from a caption + segmentation map. 2) Editing Task A: Generate an image from a segmentation map + natural language instruction. 3) Editing Task B: Generate a segmentation map from an image + natural language instruction.

Our framework can handle all of these tasks, but the methods of use differ.

twinkleyang1 commented 1 month ago

@jiangzeyinzi What I'm going to do is Conditional generation task, Generate an image from prompt+ segmentation map.

  1. The data set format is similar to that of scepter_img2img_hed_example.

  2. Your description is Source:FILE,Target:FILE,Prompt images/src_002.jpeg,images/tar_002.png,Convert to an edge map based on the prompt: A Seamless pattern with different cupcakes. Where your Source:FILE is the source FILE, Target:FILE is the hed edge processing file,。Your goal is to generate a file that is similar to the source file based on the hed boundary

  3. My description is Source:FILE,Target:FILE,Prompt images/src_R2407(1).png,images/tar_R2407(1).png,Convert to a segmentation map based on the prompt: disk and cup segmentation map My Source:FILE is a source FILE, and Target:FILE is a segmentation file. My goal is to generate a file similar to the source file based on segmentation conditions

jiangzeyinzi commented 1 month ago

The scepter_img2img_hed_example YAML or data format you are using is supported by another work, stylebooth, for image editing tasks.

In SCEdit, we extract image conditions through on-the-fly operation by registering ANNOTATORS, so the inputs are just the prompt and the target image, as shown in the several methods/scedit/ctr/xxx.yaml files you used earlier. The CONTROL_ANNO field in the corresponding YAML file is the operation of the condition we used.

You can achieve the required functionality by registering the ANNOTATORS of segment algo, such as Uniformer, in the modules/annotator directory, then use it in the yaml.