search

sail-sg / EditAnything

Edit anything in images powered by segment-anything, ControlNet, StableDiffusion, etc. (ACM MM)
Apache License 2.0
3.23k stars 179 forks source link

Details about training part #31

Open ukaukaaaa opened 1 year ago

ukaukaaaa commented 1 year ago

Hi, thanks for this great project. I have some questions about the training part.

  1. From data_build.py, we know that there is one txt file and a bunch of json files, I wonder if you can release an example of both txt and json files.
  2. Is the updated checkpoint model trained by current released training code? If not, would you mind to release the newest training code?
  3. If possible, can you complement more training details in README?
gasvn commented 1 year ago

1.The txt and json files can be refer to https://github.com/lllyasviel/ControlNet 2.The latest verision is trained with laion dataset, so the training code is a little bit differet, we will release it soon. 3.More training details will be added.

ukaukaaaa commented 1 year ago

1.The txt and json files can be refer to https://github.com/lllyasviel/ControlNet 2.The latest verision is trained with laion dataset, so the training code is a little bit differet, we will release it soon. 3.More training details will be added.

Thanks for your reply,

Since in this project, mask image is taken as an extra source input to the model, so will that change the training part, such as the json file?

Also, if I want to add one more image as a guidance such as clothing item, is it possible to incorporate this in the training and inference framework?

alelordelo commented 1 year ago

@ukaukaaaa @gasvn Im also interested on this.

A training guide to train JSON COCO (with annotation and segmented masks) would be super helpful!

gasvn commented 1 year ago

1.The txt and json files can be refer to https://github.com/lllyasviel/ControlNet 2.The latest verision is trained with laion dataset, so the training code is a little bit differet, we will release it soon. 3.More training details will be added.

Thanks for your reply,

Since in this project, mask image is taken as an extra source input to the model, so will that change the training part, such as the json file?

Also, if I want to add one more image as a guidance such as clothing item, is it possible to incorporate this in the training and inference framework?

Sorry I missed your comments. I am not sure about the exact form of how to use "one more image" as the guidance. Maybe the biggest problem here is the training data. Also, you can check our latest version, where I achieve a training free drag item from one image into another based on reference scheme. But of course, there are some failure cases as no training is evolved. With training data, I think the current pipeline would be improved.

gasvn commented 1 year ago

@ukaukaaaa @gasvn Im also interested on this.

A training guide to train JSON COCO (with annotation and segmented masks) would be super helpful!

The training data is the same to the basic controlnet. As long as you can construct a dataloader that produce the data with jpg = item['jpg'] txt = item['txt'] hint = item['hint']， you can use this training code.

alelordelo commented 1 year ago

Thanks. Loading the data is fine, the challenge is how to do multi prompt per segmentation mask?

On 22 Jun 2023, at 05:46, Shanghua Gao @.***> wrote:

@ukaukaaaa https://github.com/ukaukaaaa @gasvn https://github.com/gasvn Im also interested on this.

A training guide to train JSON COCO (with annotation and segmented masks) would be super helpful!

The training data is the same to the basic controlnet. As long as you can construct a dataloader that produce the data with jpg = item['jpg'] txt = item['txt'] hint = item['hint']， you can use this training code.

— Reply to this email directly, view it on GitHub https://github.com/sail-sg/EditAnything/issues/31#issuecomment-1601972191, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEHDNTYOY37SEYOLTDWLGSLXMO52LANCNFSM6AAAAAAYQYY7CA. You are receiving this because you commented.