search

jekwatt / tiny_utils

Collection of useful tools (for reference), so I don't have to hunt them down every time.
0 stars 0 forks source link

Custom coalesce #13

Open jekwatt opened 3 years ago

jekwatt commented 3 years ago

Add custom coalesce function in Python and the related topics.

Should be able to coalesce correctly when the original field values are blank ("")

Note that pyjanitor coalesce function assumes there are no black values and outputs correctly where values from original field values are NaN. This works for most cases but fails when all values in the original fields are NaN or a value/values in the original fields are blank.

https://pyjanitor.readthedocs.io/reference/janitor.functions/janitor.coalesce.html

jekwatt commented 3 years ago

If you already have the dataframe, this will replace NaN with blank/empty string.

df = df.replace(np.nan, "", regex=True)
df = df.fillna('')

If you are converting DataFrame to JSON, NaN will give error so best solution is in this use case is to replace NaN with None. df1 = df.where((pd.notnull(df)), None)