search

adjidieng / DETM

MIT License
130 stars 39 forks source link

Streamin Data Sources #5

Open tarikaltuncu opened 3 years ago

tarikaltuncu commented 3 years ago

Hi, is D-ETM good for streamin data sources? How do you suggest training in mini-batches of data as it accumulates?

I also have another related question. I understand that DETM can adjust the model for concept drifts by shifting the mean and varience of topics by gaussian distribution. What if there is a particular topic that exists only (and heavily) in only one (or a few) particular time interval. Would DETM be able to detect it? Or is there an assumption that all topics exist at all time intervals?

Emekaborisama commented 3 years ago

No, from my experience, all the topics don't exist at all time.

Yes, there will surely be a topic that occurs only one in a particular time interval.

Emekaborisama commented 3 years ago

When you say streaming data source, do you mean realtime data from google sheet or SQL?

mona-timmermann commented 3 years ago

You might want to have a look at the paper on Dynamic Mixture Models by Wei et al. (2007) and similar papers on topic modeling for streaming data.