i2p evaluation result error?

Maplebb commented 5 months ago

I've tested the nudity erasure model(nudity.safetensors) using the script

if len(os.listdir(args.img_output_path)) == 4703:
        print("benchmark images already generated.")
else:
    pipe = StableDiffusionPipeline.from_pretrained(
        "/root/models--CompVis--stable-diffusion-v1-4/snapshots/133a221b8aa7292a167afc5127cb63fb5005638b",
        safety_checker=None,
        requires_safety_checker = False
        # local_files_only=~args.remote,
    )

    pipe = pipe.to("cuda")
    pipe.load_lora_weights(args.spm_path)
    pipe.scheduler = LMSDiscreteScheduler(beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear", num_train_timesteps=1000)

    benchmark = pd.read_csv("/root/benchmark/i2p_benchmark.csv")

    for idx, row in benchmark.iterrows():

        sample = pipe(
            row["prompt"],
            negative_prompt=args.negative_prompt,
            width=row["sd_image_width"],
            height=row["sd_image_height"],
            num_inference_steps=50,
            guidance_scale=row["sd_guidance_scale"],
            generator=torch.cuda.manual_seed(row["sd_seed"]),
        ).images[0]
        sample.save(f"{args.img_output_path}/{idx}_0.png")
        print(f"Image saved to {args.img_output_path}/{idx}_0.png")

And get results like(threshold=0.6)

{"FEMALE_GENITALIA_COVERED": 16, 
"FACE_FEMALE": 1368, 
"BUTTOCKS_EXPOSED": 24, 
"FEMALE_BREAST_EXPOSED": 80, 
"FEMALE_GENITALIA_EXPOSED": 12, 
"MALE_BREAST_EXPOSED": 24, 
"ANUS_EXPOSED": 0, 
"FEET_EXPOSED": 34, 
"BELLY_COVERED": 4, 
"FEET_COVERED": 130, 
"ARMPITS_COVERED": 0, 
"ARMPITS_EXPOSED": 77, 
"FACE_MALE": 202, 
"BELLY_EXPOSED": 88, 
"MALE_GENITALIA_EXPOSED": 5, 
"ANUS_COVERED": 0, 
"FEMALE_BREAST_COVERED": 131, 
"BUTTOCKS_COVERED": 4}

Is there something went wrong that resulted in different results from the paper?

renjie3 commented 5 months ago

I got similar things with infer_spm.py. The images seems didn't change much before and after

Con6924 commented 5 months ago

We set different inference-time hyper-params for nudity experiments. The facilitate_factor should be adjusted to match the results in the paper. Please refer to the supplementary materials Sec.C.1 for more details.

Con6924 / SPM

i2p evaluation result error? #5