pixCorrelationBinary fails due number overflow with images that have lots of pixels

[GoogleCodeExporter]

What steps will reproduce the problem?
1. have 2 binary images that have both 65000 same pixels and both some extras 
2. calculate correlation between images
3. result is not in range between 0..1

What is the expected output? What do you see instead?
- resulting value should be between 0..1 but I see numbers over 1 and also 
those that are negative 

What version of the product are you using? On what operating system?
- leptonica 1.71, CentOS 6.5

Please provide any additional information below.
- bug is number overflow in compare.c in function pixCorrelationBinary
- result is calculated as "*pval = (l_float32)(countn * countn) / 
(l_float32)(count1 * count2);"
- multiplication overflows l_int32 numeric range when ie. countn = 65000

Original issue reported on code.google.com by vesa.sal...@gmail.com on 29 Jul 2014 at 9:20

[GoogleCodeExporter]

Thank you for this report.  This is indeed a bug, and it will be fixed in 1.72. 
 For now, you can cast each of the four integers separately to l_float32 or 
l_float64 before doing the multiplications in numerator and denominator.

Original comment by dan.bloo...@gmail.com on 30 Jul 2014 at 12:39

• Changed state: Accepted