Evaluation - Reconstruction - eval_rgb.py

[guibo-tech]

Error incompatibility with images and gt_images If there are 45 images and 45 gt_images, we get the following error. Running the code 45 images and 43 gt_images are loaded. To solve it, just copy two images more in the gt_images folder.

The same error we see to train the EndoNeRF, we have to copy two images more in the folder masks

[guibo-tech]

GPU runs out of memory, using CPU solve it,

[guibo-tech]

1st evaluation using the new generated images, all gray, why? reco_0003_training_2

python3 EndoNeRF/eval_rgb.py --gt_dir /home/guibo/repos/reco_0003/images --mask_dir /home/guibo/repos/reco_0003/gt_masks --img_dir logs/reco_0003_training_2/renderonly_path_fixidentity_000000/estim/

MSE: 0.05545433238148689 PSNR: 12.56064510345459 SSIM: 0.6856867074966431 LPIPS: 0.50532066822052

[guibo-tech]

reco_0003_training_2

python3 EndoNeRF/eval_rgb.py --gt_dir /home/guibo/repos/reco_0003/images --mask_dir /home/guibo/repos/reco_0003/gt_masks --img_dir /home/guibo/repos/BIMAP-EndoNeRF/EndoNeRF/logs/reco_0003_training_2/frames_reco_0003_training_2_fixidentity_100000_time/estim

MSE: 0.0005877256044186652 PSNR: 32.30825424194336 SSIM: 0.889693021774292 LPIPS: 0.276197612285614

[guibo-tech]

reco_0003_training_1

python3 EndoNeRF/eval_rgb.py --gt_dir /home/guibo/repos/reco_0003/images --mask_dir /home/guibo/repos/reco_0003/gt_masks --img_dir /home/guibo/repos/frames_reco_0003_training_1_fixidentity_100000_time/estim

MSE: 0.0005817899946123362 PSNR: 32.35233688354492 SSIM: 0.8909807801246643 LPIPS: 0.27124789357185364

[guibo-tech]

Comparing the two reconstructions.

In both cases, the differences in the metrics between Training 1 and Training 2 are relatively small. Prioritizing traditional image quality metrics like PSNR and SSIM, Training 1 has a slight advantage. On the other hand, prioritizing perceptual similarity, Training 1 has a slightly lower LPIPS.

1. MSE (Mean Squared Error):

   • Training 1: 0.000582
   • Training 2: 0.000588
   • Lower MSE indicates lower error and higher image quality. Training 1 has a slightly lower MSE.

2. PSNR (Peak Signal-to-Noise Ratio):

   • Training 1: 32.35
   • Training 2: 32.31
   • Higher PSNR indicates higher image quality. Training 1 has a slightly higher PSNR.

3. SSIM (Structural Similarity Index):

   • Training 1: 0.891
   • Training 2: 0.8907
   • Higher SSIM indicates greater structural similarity. Training 1 has a slightly higher SSIM.

4. LPIPS (Learned Perceptual Image Patch Similarity):

   • Training 1: 0.271
   • Training 2: 0.276
   • Lower LPIPS indicates higher perceptual similarity. Training 1 has a slightly lower LPIPS.

[guibo-tech]

The updated results you've provided are as follows:

• MSE (Mean Squared Error): 0.055
• PSNR (Peak Signal-to-Noise Ratio): 12.56
• SSIM (Structural Similarity Index): 0.686
• LPIPS (Learned Perceptual Image Patch Similarity): 0.505

These results indicate the quality and similarity of the estimated images compared to the ground truth images. Here's a brief summary of each metric:

1. MSE (Mean Squared Error): MSE measures the average squared difference between the estimated and ground truth images. A lower MSE indicates higher image quality. An MSE of 0.055 suggests relatively low error or distortion in the estimated images.

2. PSNR (Peak Signal-to-Noise Ratio): PSNR measures image quality based on the ratio of the peak signal (maximum possible value) to noise. A higher PSNR indicates higher image quality. A PSNR of 12.56 suggests that there is still room for improvement in image quality.

3. SSIM (Structural Similarity Index): SSIM measures the structural similarity between two images. A higher SSIM value indicates that the two images are more similar in terms of structure and content. An SSIM of 0.686 suggests that the estimated images are somewhat similar to the ground truth.

4. LPIPS (Learned Perceptual Image Patch Similarity): LPIPS is a perceptual similarity metric. A lower LPIPS value indicates higher similarity. An LPIPS of 0.505 suggests that there is room for improvement in terms of perceptual similarity between the estimated and ground truth images.

These metrics provide quantitative insights into the quality and similarity of the images, but the interpretation of these results depends on the specific application and the desired level of image quality. Depending on your use case, you may want to further fine-tune your model to improve these metrics or assess the visual quality of the estimated images.