topepo
commented
6 years ago Thanks for the feedback. this is very helpful. Correct me if I misinterpreted your comments.
I think that some time should be spent on engineering features with a specific outcome in mind. For example, you cover PCA, but not LDA.
We do talk about PLS but should mention that there are myriad other ways, including LDA, to do supervised dimension reduction.
There are a lot more 1:1 transformations that I imagine your readers would want to know about. see: http://www.deducer.org/pmwiki/pmwiki.php?n=Main.TransformVariables
Thanks for the link. I should add something about using quantiles too.
I'm not a big fan of hashing categorical variables, though it definitely should be covered.
Me either. See section 5.2. This is dying for a good statistical solution.
Another option when you have a specific outcome (and a prediction algorithm that needs dimension reduction) is to fit a simple hierarchical model e.g....
That's good. I'm on the fence about including it since it brings a lot of other machinery with it (related to the estimation procedure). Is there a reference for this?
On a related note, you should mention the option of including dummy codes for the c most common categories and an 'other' bucket for the rest.
This should probably be a whole chapter alone and deal with bag-of-words, co-occurance, sentiment analysis, word vectors, stemming, tokenizing, etc.
What about image data? e.g. using the (intermediate) output of a general purpose NN model (imagenet), gabor filters, etc.
There is some text-related stuff in section 5.5 but that is a whole book unto itself.
We don't do a lot with image and text here because that is what 95% of the references with "feature engineering" in their titles talk about. I'd rather spend time on topics with less coverage.
What about feature extraction from network data? e.g. number of friends, shared partners, etc.
Good idea but I've never done any of that so I wouldn't be qualified to talk about it.
You sort of assume that you've already got your data in a flat file. How about dealing with engineering out of a structure DB (e.g SQL)? If you have an outcome in one table, how do you extract relevant features from other tables if there is a one-to-one, many-to-one, one-to-many or many-to-one relationship?
There will be some of that in chapter 8 (Flattening Profile Data) with some examples.
Shouldn't there be a discussion of what transformations are needed for what types of models? e.g. Don't code interactions for an RF. Do scale your predictors if your fitting algorithm is gradient descent.
Yes. There will be something like Appendix A in Applied Predictive Modeling. I'm not sure where it goes just yet though.
How should outliers be handled? univariate and multivariate. The choice between row exclusion and column transformation.
There is some of that in section 6.3.3 but it could use some expansion.
ifellows
rwarnung
alexpghayes
rodfloripa
rkb965
Great idea for a book. So much of an analyst's time is spent here, and no real resources are available treating it in a comprehensive manner.
I did a quick skim over your work so far. Great job! here are a few thoughts:
I think that some time should be spent on engineering features with a specific outcome in mind. For example, you cover PCA, but not LDA.
There are a lot more 1:1 transformations that I imagine your readers would want to know about. see: http://www.deducer.org/pmwiki/pmwiki.php?n=Main.TransformVariables
I'm not a big fan of hashing categorical variables, though it definitely should be covered. Another option when you have a specific outcome (and a prediction algorithm that needs dimension reduction) is to fit a simple hierarchical model e.g.:
$$ y_{ij} = \thetajX{ij} + \epsilon_{ij} $$ where $$ \theta_j \sim N(0, \sigma). $$ Here X is the dummy coded categorical variable. Then use \theta as the encoded value for X. This shrinks rare categories toward 0 so they don't have a lot of impact, and puts the rest on the right scale for a regression. Often I find that interactions with this extracted feature are important.
On a related note, you should mention the option of including dummy codes for the c most common categories and an 'other' bucket for the rest.
Feature engineering is perhaps most important in dealing with text, where the raw data is rarely if ever analyzed directly. This should probably be a whole chapter alone and deal with bag-of-words, co-occurance, sentiment analysis, word vectors, stemming, tokenizing, etc.
What about feature extraction from network data? e.g. number of friends, shared partners, etc.
You sort of assume that you've already got your data in a flat file. How about dealing with engineering out of a structure DB (e.g SQL)? If you have an outcome in one table, how do you extract relevant features from other tables if there is a one-to-one, many-to-one, one-to-many or many-to-one relationship?
What about image data? e.g. using the (intermediate) output of a general purpose NN model (imagenet), gabor filters, etc.
Shouldn't there be a discussion of what transformations are needed for what types of models? e.g. Don't code interactions for an RF. Do scale your predictors if your fitting algorithm is gradient descent.
How should outliers be handled? univariate and multivariate. The choice between row exclusion and column transformation.