is the traindata model input correct？

[lawsonxwl]

Describe the bug

I try to fine tune the official sdxl brushnet with my own data, but get bad result, I wonder if my input is correct, below is my ground truth,mask and masked image. Masked image latent concat resized mask and they are feed into brushnet as condition, also the noised ground truth latent is feed into brushnet too.

Setting: 2*A100 bs = 8 Gradient accumulation =2 lr = 1e-5 Steps = 30000

finetune result A man standing by the sea Before ft: After 30000 steps ft

Reproduction

/

Logs

No response

System Info

/

Who can help?

No response

[Shuvo001]

@lawsonxwl you can check it out https://github.com/TencentARC/BrushNet/issues/35#issuecomment-2177491312

[lawsonxwl]

@lawsonxwl you can check it out #35 (comment)

thanks for the reply, i have checked that, could you show me your generation result? I have trained SDXL ver brushnet from scratch, the generation result is not so good but also, not so bad.. according to what u say in that issue, for the mask, foreground area is set to 1, background is 0, so the inpaint area is 0 and keep area is 1?

[Shuvo001]

@lawsonxwl

here is some outputs. my input image:

some outpainting results from my burshnet sdxl trained model:

NOTE: i used LoRA for my needs.

[lawsonxwl]

@lawsonxwl

here is some outputs. my input image:

some outpainting results from my burshnet sdxl trained model:

NOTE: i used LoRA for my needs.

that's good，you use juggernaut_X for training or original sdxl? when using lora, what module do you train? do you train base model and brushnet together? could you tell me the batch size, training data and lr you use? thanks for the reply!

[lawsonxwl]

@lawsonxwl

here is some outputs. my input image:

some outpainting results from my burshnet sdxl trained model:

NOTE: i used LoRA for my needs.

btw, could you tell me if my input is right? for mask inpaint area is 0 and reserve area is 1, for masked_image inpaint area is 0(black)

[Shuvo001]

@lawsonxwl

1. i used my own fine tuned sdxl model for testing, original model is ok as well u can try.

2. my hyperparameters:

3. used custom dataset.

4. for mask input its not 0 and 1. you can follow this: https://github.com/TencentARC/BrushNet/issues/35#issuecomment-2177908586

[lawsonxwl]

As u say in the comment:

The resulting output of the rle2mask function is mainly a binary mask image where the ones represent the foreground pixels and zeros represent the background pixels. U say foreground pixel is one and background pixel is zero! Btw could I add your wechat so we can communicate easier?

发自我的iPhone

------------------ Original ------------------ From: RxxS @.> Date: Tue,Jun 25,2024 10:22 AM To: TencentARC/BrushNet @.> Cc: lawsonxwl @.>, Mention @.> Subject: Re: [TencentARC/BrushNet] is the traindata model input correct？ (Issue #44)

@lawsonxwl

i used my own fine tuned sdxl model for testing, original model is ok as well u can try.

my hyperparameters: image.png (view on web)

used custom dataset.

for mask input its not 0 and 1. you can follow this: #35 (comment)

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[zjlinkin]

may i add your wechat? i'm facing the same problem as you

[lawsonxwl]

thegreatlsx

发自我的iPhone

------------------ Original ------------------ From: zjlinkin @.> Date: Wed,Jul 3,2024 11:47 AM To: TencentARC/BrushNet @.> Cc: lawsonxwl @.>, Mention @.> Subject: Re: [TencentARC/BrushNet] is the traindata model input correct？ (Issue #44)

may i add your wechat? i'm facing the same problem as you

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>