jmoralez
commented
2 years ago WDYT about using map on the dicts instead?
import numpy as np
import pandas as pd
nrows = 1_000_000
ncols = 20
n_unique_vals = 100
df = pd.DataFrame(
np.random.randint(0, n_unique_vals, (nrows, ncols)),
columns=[f'x{i}' for i in range(ncols)],
dtype=str
)
cols_to_replace = [f'x{i}' for i in range(0, 20, 2)]
vec = {col: {str(i): str(i + 1) for i in range(n_unique_vals - 10)} # last 10 values were not seen
for col in cols_to_replace}
# define one of the replacement columns as float
df['x0'] = df['x0'].astype('float')
vec['x0'] = {float(k): float(v) for k, v in vec['x0'].items()}
replace_unseen = -1
def apply_replacements(df, columns, vec, replace_unseen):
def column_categorizer(col: str):
return np.select(
# the original had an and here so I guess it should be &
[df[col].isna() & (df[col].dtype == "float"), ~df[col].isin(vec[col].keys())],
[np.nan, replace_unseen],
df[col].replace(vec[col])
)
return df.assign(**{col: column_categorizer(col) for col in columns})
%time res1 = apply_replacements(df, cols_to_replace, vec, replace_unseen)
# Wall time: 1min 22s
# proposal
def apply_replacements2(df, columns, vec, replace_unseen):
def column_categorizer(col: str):
replaced = df[col].map(vec[col])
unseen = df[col].notnull() & replaced.isnull()
replaced[unseen] = replace_unseen
return replaced
return df.assign(**{col: column_categorizer(col) for col in columns})
%time res2 = apply_replacements2(df, cols_to_replace, vec, replace_unseen)
# Wall time: 3.93 s
pd.testing.assert_frame_equal(res1, res2)
matheusfacure
codecov-commenter
Status
IN DEVELOPMENT
Todo list
Background context
Pandas .apply is incredibly slow to run. Since this function is used in multiple learners, speeding it up should yield tremendous grains in performance. As a side note, we should never introduce new call
.apply. They are a major source of headacheDescription of the changes proposed in the pull request
Vectorize
apply_replacements