Although it is bad practice, It is common to use a "placeholder value" to represent a missing value. For example, in a continuous column the value "999" or "-1" can be used to represent missing data.
The current implementation of SDV can handle null values, but it only identifies missing values by data.isnull() in the NullTransformer. Hence, the user needs to transform missing values such as the ones mentioned above before sending it to SDV.
Expected behavior
It would be convenient if SDV could extend its metadata file to allow custom null values per column. This data could later be used to transform the custom null values to np.nan before reaching the NullTransformer.
Thanks for the request @MLjungg. A few follow up questions that will help us prioritize the feature:
In your experience, is there typically a singular placeholder value or can there be multiple values that mean different things? Eg 999 and -1 can both be present but have different meanings
How are you planning to use the synthetic data? Is it your expectation that the same custom missing values will appear in the synthetic data too?
Problem Description
Although it is bad practice, It is common to use a "placeholder value" to represent a missing value. For example, in a continuous column the value "999" or "-1" can be used to represent missing data.
The current implementation of SDV can handle null values, but it only identifies missing values by
data.isnull()
in theNullTransformer
. Hence, the user needs to transform missing values such as the ones mentioned above before sending it to SDV.Expected behavior
It would be convenient if SDV could extend its metadata file to allow custom null values per column. This data could later be used to transform the custom null values to
np.nan
before reaching theNullTransformer
.