search

JihyongOh / DeMFI

[ECCV 2022] Official repository of DeMFI.
79 stars 8 forks source link

Questions about the results #8

Open VongolaWu opened 2 years ago

VongolaWu commented 2 years ago

May I ask some questions about the test results? Could you explain the meanings of the names: Time, intp_testPSNR_prime, deblur_testPSNR_prime, intp_testPSNR, deblur_testPSNR.

Does this 'Time' represent the whole time for deblurring and interpolating 8 different images or for 1 image? What is the difference between 'prime' and not prime? Also, why are the results always followed by an (avg)? What is it?

For example: image

JihyongOh commented 2 years ago

@VongolaWu Thank you for your interest.

  1. 'prime' means the performances for the architecture of DeMFI-Netbs(first stage)'s outputs, w/o 'prime' means the performances for those of DeMFI-Netrb(second stage)'s.

  2. 'Time' represents an inference time (runtime) for one image.

  3. Performance results are shown based on the total average for each time index (1~7: intermediate time index for joint deblurring and interpolation [MFI x8], 8: time index for only deblurring).

VongolaWu commented 2 years ago

I still feel a little confused. Could you further explain it: 1

The part below "------x8 MFI results----" should be the final result, right? Using the above image as an example, I have already split it as 5 parts: "Test after Epoch", "x8 MFI results[PSNR Stage 1]", "x8 MFI results[SSIM Stage 1]", "x8 MFI results[PSNR Stage 2]", "x8 MFI results[SSIM Stage 2]"

  1. What is the relationship between "Test after Epoch" and the other 4 parts? Is the "Test after Epoch" whole report for all the test images? But I'm still confused about the (avg:) after each item. For example, "intp_testPSNR: 2.6699e+01 (avg:2.3329e+01)", what does the avg mean?
  2. Also I still feel confused about the (avg) in "x8 MFI results[PSNR Stage 1]", "x8 MFI results[SSIM Stage 1]", "x8 MFI results[PSNR Stage 2]", "x8 MFI results[SSIM Stage 2]".
  3. Does "Time" mean that generating 8 sharp images for 1 blur image needs 1.136 seconds?
JihyongOh commented 2 years ago

@VongolaWu

  1. Yes, first block is average of all results for all time indices. "avg:~" means that it shows literally average on that time index (in your example, "avg:2.3329e+01" is a final performance for intp_testPSNR).
  2. The final DeMFI-Net_{rb} (Stage II) performance is based on "avg" values of "x8 MFI results[PSNR Stage 2]" and "x8 MFI results[SSIM Stage 2]" for each time index (as in Fig. 1). Therefore, the "Average" performance in Table is calculated by averaging all "avg" values for all time indices of "x8 MFI results[PSNR Stage 2]", "x8 MFI results[SSIM Stage 2]" for each metric.
  3. No, "Time" means an average runtime for running DeMFI-Net once, i.e. synthesizing each pair of three frames (two deblurred frame - S0, S1 and one interpolated image - St)
VongolaWu commented 2 years ago

Thank you for your explanation. Let me ask a further question: image I make a table for this result. image Therefore, for the first table, the intp_testPSNR(avg) should be the final results for 7 interpolated frames? and deblur_testPSNR(avg) should the final results for the 1 deblurred frames? But they are not consistent with the second table. So I would like to ask if I would like to represent the average PSNR for the whole test dataset containing both interpolated frames and deblurred frames, which number should I use?

JihyongOh commented 2 years ago

@VongolaWu First table is for an average of "whole" samples, regardless for each scene's numbers of frames. Second table shows an average of "average for each scene", as done in BIN paper's evaluation. It considers that the numbers of frames are different for each scene. Therefore, it is recommended to use the second table format to be robust to imbalance of numbers of frames per each scene. (in your case, average 30.8135 dB to evaluate joint performance)

VongolaWu commented 2 years ago

@JihyongOh Thank you for your reply. One more thing: For PSNR_1 to PSNR_7, are they the generated frame at timestamp t*1/8 (eg. PSNR_1 is at time 1/8)? And the PSNR_blur is the sharp frame of B0?

JihyongOh commented 2 years ago

@VongolaWu Yes, your understanding for PSNR_1 (1/8) to PSNR_7 (7/8) is right (multiple of 1/8), and PSNR_blur is measured for the sharp frame of S1 and S1 without overlapped evaluation by considering a temporal window. Please refer these two lines; line1 and line2.