search

AbeHandler / rookie

The Rookie Text Analysis System
10 stars 1 forks source link

Text compression in javascript? #143

Closed AbeHandler closed 8 years ago

AbeHandler commented 8 years ago

Pushing large numbers of tokens between client and server is expensive. Maybe it makes sense to compress (at least common) tokens and deserialize on the client side? (ex: "the" = 0, "or" = 1, "and"= 01 etc.) This might be necessary for handling more text in the browser.

http://nytimes.github.io/pourover/ http://nytimes.github.io/tamper/

brendano commented 8 years ago

in my experience, turning on gzip compression at the http protocol level is a huge advantage compared to doing nothing (5x savings at least), and it's very easy to do. it's not reassuring that the tamper website doesn't show any space comparisons.

AbeHandler commented 8 years ago

@brendano I guess I was thinking about this less for Rookie than for interactive NLP in the browser in general. Maybe something analogous to tamper/pourover would be useful? Although I guess one should probably have at least a single concrete use case before dreaming up giant frameworks.