step_lump_* steps

[EmilHvitfeldt]

Way to combine levels in factor variables. I will henceforth be calling this action lumping.

There are two main ways we can go about doing this:

• target/response agnostic
• target/response based

Target agnostic

• stringdist
• forcats wrappers around fct_lump_min(), fct_lump_prop(), fct_lump_n(), fct_lump_lowfreq()
• hashing trick (technically included on the list, but won't be included since the trained step would be function based rather then list based)
• hierarchical clustering
• cluster in word embedding

Target based

• tree-based. Fit a tree model using the factor and the response. lump levels together according to the leaves

References

https://arxiv.org/pdf/2104.00629.pdf

[EmilHvitfeldt]

Think a lot about this should work on characters/factors. How should factor levels be handled

[EmilHvitfeldt]

think about keyword "lump/collapse"

[EmilHvitfeldt]

use fct_collapse() internally to update the levels

library(forcats)

dict <- list(missing = c("No answer", "Don't know"),
             other = "Other party",
             rep = c("Strong republican", "Not str republican"),
             ind = c("Ind,near rep", "Independent", "Ind,near dem"),
             dem = c("Not str democrat", "Strong democrat"))

fct_collapse(gss_cat$partyid, !!!dict)