search

JasonKessler / scattertext

Beautiful visualizations of how language differs among document types.
Apache License 2.0
2.23k stars 289 forks source link

Caching results for update #39

Open NickyFot opened 5 years ago

NickyFot commented 5 years ago

Your Environment

NickyFot commented 5 years ago

Looks like if we have a corpus that updates over time (eg. Twitter feed) we have to rerun the code for everything (eg. st.produce_scattertext_explorer()) We can pickle objects and reload in the future, so maybe a method to run explorer on new items only? it would significantly lower processing time on each update.

JasonKessler commented 5 years ago

Hi Nicky,

Appreciate the feedback.

Are you able to quickly build new Corpus objects when there are updates to documents? It would be somewhat straightforward to have add, say, an "add_documents(parsed_documents, categories)" method to the TermDocMatrix family of classes.

Unfortunately, produce_scattertext_explorer uses term positioning, scoring, and selection methods which can depend the distribution or rank of term frequencies. This means that adding a single document can change the scores or positions of every other term, and makes it very difficult to update the visualization without completely regenerating it.

However, if you'd like to give either of these a shot, I'd be happy to add your improvements to the codebase.

Jason