search

araddon / httpstream

A client for http streaming API's in Go: Twitter, Flowdock, DataSift, Custom
62 stars 20 forks source link

Using delimited to increase performance? #6

Open lateefj opened 10 years ago

lateefj commented 10 years ago

Wasn't sure if using delimited would improve performance but it would allow for just reading the exact number of bytes. Docs: https://dev.twitter.com/docs/streaming-apis/parameters#delimited. I noticed that Python implementation https://github.com/tweepy/tweepy/blob/master/tweepy/streaming.py uses and so I looked into it. Thoughts?

araddon commented 10 years ago

This is probably not a bad idea, currently it scans for new lines https://github.com/araddon/httpstream/blob/master/stream.go#L174 , doubt the impact would be noticeable unless doing many more than currently max available from twitter stream (3050/min). The underlying connections, buffering, re-connections all probably could be refactored and cleaned up. So, ill leave it as an open issue for now.

lateefj commented 10 years ago

True, I noticed the tweepy implementation to be streaming a lot faster but I would think this is because of some other issue not the Reader.