compress() was setting the low order bit instead of the high. From RFC 8032:
To form the
encoding of the point, copy the least significant bit of the
x-coordinate to the most significant bit of the final octet.
decompress() was incorrectly choosing the x value. While in curve25519-dalek the sqrt_ratio_i always returns the nonnegative square root, this is not true for this crate's sqrt_ratio.
Otherwise, if x_0 != x mod 2, set x <-- p - x
decompress() did not handle correctly the identity point. sqrt_ratio returned is_err = 1 for it even if the sqrt is zero, which seems to be caused by the trick of multiplying by u (which does not work if it's 0). I think the implementation could follow RFC 8032 more closely in that part, but I wanted to get the simplest fix first.
Fix these bugs:
compress()
was setting the low order bit instead of the high. From RFC 8032:decompress()
was incorrectly choosing the x value. While in curve25519-dalek thesqrt_ratio_i
always returns the nonnegative square root, this is not true for this crate'ssqrt_ratio
.decompress()
did not handle correctly the identity point.sqrt_ratio
returnedis_err = 1
for it even if the sqrt is zero, which seems to be caused by the trick of multiplying byu
(which does not work if it's 0). I think the implementation could follow RFC 8032 more closely in that part, but I wanted to get the simplest fix first.