Can't generate images base on images correct

[illrayy]

Originally posted by @yangjie1874 in https://github.com/UTokyo-FieldPhenomics-Lab/DODA/issues/2#issuecomment-2209887718

Hello, as shown in the figure, in the image generated by generate_data_for_target_domain.py, the label position I want is very random and almost wrong. I want to know the reason

[illrayy]

@yangjie1874 It seems that your controlnet has not converged, or the Control Strength is set too small. This is the image I got with the pre-trained DODA. For controlnet, how many iterations did you train?

[yangjie1874]

sorry, I using ldm, I have no training controlnet,

[yangjie1874]

just train ldm in the same way as #2

[illrayy]

DODA uses label (layout images in paper) to control the image generation, and the control achieved through controlnet(cldm), so if you only use ldm, it won't use layout images to control the generation, and the label won't match the image.

Training cldm requires boxes annotation, if you don't have the annotation, you can directly use the pre-trained weight to generate data. OR mix your data with GWHD to train the ldm first, then use the annotation from GWHD to train the cldm. As show in Table 3 and Figure 5 of the paper, although there is no annotation from the target domain, this training method can still improve the quality of the generated target domain data.

[yangjie1874]

Thank you very much for your patient answer. You mean I will now use the trained ldm as pre-trained weights to train cldm?

[illrayy]

for your dataset, do you have annotation?

[yangjie1874]

yes！

[yangjie1874]

The ldm I trained is unlabeled, but when I use the weights of ldm to train cldm, an error occurs. what should i do ? /(ㄒoㄒ)/~~

[yangjie1874]

Or maybe I didn't notice some problems when I was training ldm?

[illrayy]

yes！

This work is designed for unlabeled scenarios, and given the size of your dataset, I'm not sure if it would serve your purpose

The ldm I trained is unlabeled, but when I use the weights of ldm to train cldm, an error occurs. what should i do ? /(ㄒoㄒ)/~~

DId you run tool_add_wheat_control.py to add the controlnet? If not, please change the input_path and output_path in the file, then run the file