chaunceywx
commented
4 months ago Hi, thanks for the great work and the comprehensive evaluation framework. We recently wanted to add your work for comparison but found that there seems to be a problem in evaluating the quality of generated images. Please tell me if there is any misunderstanding:
As line 233 in scripts/pose2img_metrics.py shows, we saved the images as a concatenation of four images: [generated_image, pose_image, text_image, original_image]. However, in your implementation of the quality evaluation in utils/metrics/evalute_metrics_of_each_category.py, you seem to load the entire image for evaluation rather than splitting out the generated image. Please refer to line 78. We believe this will lower down the quality scores.
We tried to revise the codes in evalute_metrics_of_each_category.py by adding three lines of codes between line 77 and line 78:
img = np.array(img).reshape(512,4,512,3) \\ img = cv2.cvtColor(img[:,0,:,:], cv2.COLOR_BGR2RGB) \\ img = Image.fromarray(img)The codes above basically extracts the generated image from the saved image. Having revised this, we are able to achieve a FID of about 10 using your pretrained checkpoint (much lower than reported).Thanks and I'm looking forward to further discussion :)
Do you calculate metrics through:
python scripts/pose2img_metrics.py --outdir outputs/metrics --config utils/metrics/metrics.yaml --ckpt humansd_data/checkpoints/humansd-v1.ckpt
??
I calculate metrics in this way but encountered this error:
Loading model from humansd_data/checkpoints/higherhrnet_w48_humanart_512x512_udp.pth
Traceback (most recent call last):
File "E:\code\T2F\HumanSD-main\scripts\pose2img_metrics.py", line 260, in <module>
main()
File "E:\code\T2F\HumanSD-main\scripts\pose2img_metrics.py", line 151, in main
model = load_model_from_config(config, f"{opt.ckpt}")
File "E:\code\T2F\HumanSD-main\scripts\pose2img_metrics.py", line 57, in load_model_from_config
model = instantiate_from_config(config.model)
File "F:\conda_env\envs\HumanSD_21\lib\site-packages\omegaconf\dictconfig.py", line 353, in __getattr__
self._format_and_raise(
File "F:\conda_env\envs\HumanSD_21\lib\site-packages\omegaconf\base.py", line 190, in _format_and_raise
format_and_raise(
File "F:\conda_env\envs\HumanSD_21\lib\site-packages\omegaconf\_utils.py", line 821, in format_and_raise
_raise(ex, cause)
File "F:\conda_env\envs\HumanSD_21\lib\site-packages\omegaconf\_utils.py", line 719, in _raise
raise ex.with_traceback(sys.exc_info()[2]) # set end OC_CAUSE=1 for full backtrace
File "F:\conda_env\envs\HumanSD_21\lib\site-packages\omegaconf\dictconfig.py", line 351, in __getattr__
return self._get_impl(key=key, default_value=_DEFAULT_MARKER_)
File "F:\conda_env\envs\HumanSD_21\lib\site-packages\omegaconf\dictconfig.py", line 438, in _get_impl
node = self._get_node(key=key, throw_on_missing_key=True)
File "F:\conda_env\envs\HumanSD_21\lib\site-packages\omegaconf\dictconfig.py", line 470, in _get_node
raise ConfigKeyError(f"Missing key {key}")
omegaconf.errors.ConfigAttributeError: Missing key model
full_key: model
object_type=dictIt looks like configs in the metrics.yaml are incomplete. What configs should I add to utils/metrics/metrics.yaml? Can you give me some advice?
Hi, thanks for the great work and the comprehensive evaluation framework. We recently wanted to add your work for comparison but found that there seems to be a problem in evaluating the quality of generated images. Please tell me if there is any misunderstanding:
As line 233 in scripts/pose2img_metrics.py shows, we saved the images as a concatenation of four images: [generated_image, pose_image, text_image, original_image]. However, in your implementation of the quality evaluation in utils/metrics/evalute_metrics_of_each_category.py, you seem to load the entire image for evaluation rather than splitting out the generated image. Please refer to line 78. We believe this will lower down the quality scores.
We tried to revise the codes in evalute_metrics_of_each_category.py by adding three lines of codes between line 77 and line 78:
img = np.array(img).reshape(512,4,512,3) \\ img = cv2.cvtColor(img[:,0,:,:], cv2.COLOR_BGR2RGB) \\ img = Image.fromarray(img)The codes above basically extracts the generated image from the saved image. Having revised this, we are able to achieve a FID of about 10 using your pretrained checkpoint (much lower than reported).Thanks and I'm looking forward to further discussion :)