search

snuspl / cruise

Cruise: A Distributed Machine Learning Framework with Automatic System Configuration
Apache License 2.0
26 stars 2 forks source link

Consistency issue in LDA model during worker-side data migration #1222

Open wynot12 opened 7 years ago

wynot12 commented 7 years ago

When migrating worker-side data in LDA app, we have a problem that model is damaged during migration. It's because some of our apps have also partial models in the input table.

However, ET/EM assumes that worker-side table has only immutable input data. As a result, we design them not to care about worker-side table data consistency.

To resolve this problem, we need to provide a consistency mechanism for worker-side tables. Or we may change LDA app to not to use local cache or cope with damage in local model.

wynot12 commented 6 years ago

Need to fix it, since LDA is a primary workload in multi-job situation.