Open asfimport opened 1 year ago
Gang Wu / @wgtmac: These are good questions. Let me try to answer them from the perspective of Java:
Java does not have signaling NaN. And it only has a single NaN representation. https://stackoverflow.com/a/25051746
To support better interoperability, I think we should do two things:
Gabor Szadovszky / @gszadovszky: Bloom filters are for searching for exact values. Exact checking of floating point numbers are usually code smell. Usually checking if the difference is below an epsilon value is suggested over using exact equality. I am wondering if there is a real usecase for searching for an exact floating point number. Maybe disabling bloom filters completely for FP numbers is the simplest choice and probably won't bother anyone.
If we still want to handle FP bloom filters I agree with @wgtmac's proposal. (It is a similar approach we implemented for min/max values.) Keep in mind that we need to handle the case when someone wants to filter on a NaN.
Gang Wu / @wgtmac: I think there is a similar issue in the dictionary encoding of floating-point types. @gszadovszky
Gabor Szadovszky / @gszadovszky: But we don't build the dictionary for filtering but for encoding. We should not add anything else than what we have in the pages. So anything should be added to the read path.
Maybe we do not need to handle +0.0 and -0.0 differently from the other values. (We needed to handle them separately for min/max values because the comparison is not trivial and there were actual issues.) If someone deals with FP numbers they should know about the difference between +0.0 and -0.0.
Because the FP spec allows to have multiple NaN values (even though java use one actual bitmap for it) we need to avoid using Bloom filter in this case. Dictionary is a different thing because we deserialize it to java Double/Float values in a Set so we will have one NaN value that is the very same one we are searching for. (It is more for the other implementations to deal with NaN if the language has several NaN values.)
Currently, our Parquet can use BloomFilter for any physical types. However, when BloomFilter apply on float:
Should qNaN sNaN written in BloomFilter? Are they equal?
Reporter: Xuwei Fu / @mapleFU
Note: This issue was originally created as PARQUET-2255. Please see the migration documentation for further details.