linkedin / detext

DeText: A Deep Neural Text Understanding Framework for Ranking and Classification Tasks
BSD 2-Clause "Simplified" License
1.26k stars 133 forks source link

Add embedding and MLP support for sparse wide features #24

Closed StarWang closed 4 years ago

StarWang commented 4 years ago

Description

Currently DeText's design for sparse feature has simple modeling power for sparse features.

  1. only linear model is applied on sparse features
  2. there's no interaction between sparse features and dense features (model_score = dense_score + sparse_score)

This PR resolves the above limitation on sparse feature by

  1. computing dense representation of sparse features
  2. allowing interactions between sparse features and wide features

More specifically, the model architecture changes from

dense_score = dense_ftrs -> MLP
sparse_score = sparse_ftrs -> Linear
final_score = dense_score + sparse_score

to

sparse_emb_ftrs = sparse_ftrs -> Dense(sp_emb_size)
all_ftrs = (dense_ftrs, sparse_emb_ftrs) -> Concatenate
final_score= all_ftrs -> MLP

Type of change

List all changes

Please list all changes in the commit.

Testing

zhoutong-fu commented 4 years ago

Two general comments: 1) in either the description or readme file update the latest model structure (can use the new feature Xiaowei added last time that dumps the structure to a txt file; 2) can you check if run_ detext_multitask.sh works on multitask test data (not the sparse wide features test data)?

StarWang commented 4 years ago

Two general comments: 1) in either the description or readme file update the latest model structure (can use the new feature Xiaowei added last time that dumps the structure to a txt file; 2) can you check if run_ detext_multitask.sh works on multitask test data (not the sparse wide features test data)?

@zhoutong-fu Thanks for the review! I've updated the readme file and tested that run_detext_multitask.sh runs successfully