linkedin / photon-ml

A scalable machine learning library on Apache Spark
Other
790 stars 185 forks source link

Union random effect prior model coefficients only for incremental tra… #476

Closed cmjiang closed 3 years ago

cmjiang commented 3 years ago

The prior models of random effect are used to join the random effect dataset and then to get the union of new features (in the dataset) and old features (in the prior model). This is useful in incremental training, but costs inefficiency in other jobs since "join" is expensive. In this PR, we change this join-and-union operation only for incremental training.

junshi15 commented 3 years ago

LGTM.