PSNR Measure - Githubissues

tibuch commented 4 years ago

Hi,

I have a question about the PSNR Measure.

I am using from dival.measure import PSNR to compute the PSNR values on my results. Now just to be sure that everything is working I decided to compute the PSNR values of FBP to ground truth. I am using this reconstructor = construct_reconstructor('fbp', 'lodopab', impl='astra_cpu') as FBP reconstructor.

Now in the paper 'The LoDoPaB-CT Dataset' the PSNR for the FBP reconstruction of test sample 0 is given as 16.1 (Fig. 5). In the paper 'Computed Tomography Reconstruction using Deep Image Prior' the PSNR for the FBP reconstruction of test sample 0 is given as 27.3 (Fig. 8). And if I compute it in my code I get a PSNR of 14.6.

Why are these results so different?

jleuschn commented 4 years ago

Hi,

the main difference is the hyper parameters:

in 'The LoDoPaB-CT Dataset': filter_type='Ram-Lak' and frequency_scaling=1.0, which would be the exact reconstruction formula from noise-free measurements, but is clearly sub-optimal for the low-dose data
in 'Computed Tomography Reconstruction using Deep Image Prior': filter_type='Hann' and frequency_scaling=0.641, which have been determined using some validation samples

I think you were looking to use the function get_reference_reconstructor, which invokes construct_reconstructor and afterwards loads the optimized parameters. Maybe i can improve the docs about these functions or make the latter private.

Finally for the difference of 14.6 vs 16.1, this could well stem just from a different backend implementation (or version of odl/astra), e.g. i get 14.6 with 'astra_cpu' and 15.9 with 'astra_cuda'.

Hope this clarifies a bit!

tibuch commented 4 years ago

Thank you for the answer!

Okay with the get_reference_reconstructor I get a mean PSNR of 22.9 for the FBP astra_cpu implementation. Which is in the ballpark of the results reported in 'Computed Tomography Reconstruction using Deep Image Prior'. The remaining difference comes probably from the backend difference.

jleuschn commented 4 years ago

Hmm, 22.9 seems too little, i get 27.1 with 'astra_cpu'. Maybe the odl and astra versions are not compatible?

tibuch commented 4 years ago

Oh, you are right! I compared the lodopab mean PSNR to the ellipses mean PSNR..

Hmmm.. how can I make sure that astra and odl are compatible?

I have these versions:

astra-toolbox             1.9.9.dev4               py37hb48bd9c_0    astra-toolbox/label/dev
libastra                      1.9.9.dev4               cuda_10.1_0           astra-toolbox/label/dev
odl                             1.0.0.dev0               pypi_0                     pypi

jleuschn commented 4 years ago

The versions look fine, the latest of both work for me. Only thing i could imagine is an outdated odl (it's 1.0.0.dev0 for some time), but i would not think so.

Indeed the 22.9 match the ellipses PSNR well, but i assume that is just coincidence?

No idea why it does not match then :(

tibuch commented 4 years ago

Okay, I rerun everything again with cleared caches and now it looks perfectly fine:

Mean PSNR: 30.5 PSNR test sample 0: 27.1

Sorry about the confusion and thank you very much for your swift reply!

jleuschn / dival

PSNR Measure #27