features.py implements several feature-processing functions. For missing values, there are two plausible ways of embedding them into the Bloom filter:
Process into a token which is then embedded consistently into the same slots
Ignore completely, so that a missing value has no effect on the Bloom filter
The second option is preferable, because we often want a missing value to compare to a non-missing value. For example, one record may be coded as 'Female' and the other record as NA -- there's no reason to prefer missing records to have greater similarity to other records with missing values, as opposed to having equal similarity to all other records.
I think currently the preferred option is implemented on some of the feature-processing functions, but not consistently. We need to ensure it's consistent, and also implement tests to check for this handling.
features.py implements several feature-processing functions. For missing values, there are two plausible ways of embedding them into the Bloom filter:
The second option is preferable, because we often want a missing value to compare to a non-missing value. For example, one record may be coded as 'Female' and the other record as NA -- there's no reason to prefer missing records to have greater similarity to other records with missing values, as opposed to having equal similarity to all other records.
I think currently the preferred option is implemented on some of the feature-processing functions, but not consistently. We need to ensure it's consistent, and also implement tests to check for this handling.