search

parroty / extwitter

Twitter client library for elixir.
MIT License
409 stars 129 forks source link

Allow for multiple tweets in a given message #104

Closed pjskennedy closed 3 years ago

pjskennedy commented 6 years ago

I noticed internal Poison parsing errors while using the streaming API:

[error] Error returned, stopping stream ({{:invalid, "{", 8142}, "{\"created_at\":\"Sat Oct 13 23:00:18 +0000 2018\"...

Twitter's API Doc's:

note that Tweets and other streamed messages will not necessarily fall on HTTP chunk boundaries – be sure to use the delimiter defined above to determine activity boundaries when reassembling the stream.

I took a look into the code and found that sometimes detecting an "end of tweet" was not detecting there was actually two (or more) tweet JSON documents concatenated together. It looks as though each HTTP Chunk returned could have multiple tweets in it.

The line feed (\r\n) is always to delimit tweets in this API, even if it exists in a following chunk. The existing code was skipping "Empty" messages (messages that are only line feeds), however these line feeds are often marking the end of a tweet in a previous chunk in the HTTP response. This is what was leading to appended JSON documents and corresponding JSON parse failures.

coveralls commented 6 years ago

Coverage Status

Coverage increased (+0.6%) to 82.718% when pulling 3a45fe884a2bee4b8f45476762f031591d94b8fe on pjskennedy:inconsistent-chunking-streaming-api into 64c9e875afd706f9ace3e6046e8b9c6ef12d4aab on parroty:master.

jessiahr commented 5 years ago

Is there a plan to merge this? stream_filter errors out as soon as it hits an :invalid response, would be nice to not have to restart it.

KristerV commented 3 years ago

i guess i'll use the fork then. edit: oh it's got a bunch of missing commits.. hmm

KristerV commented 3 years ago

I've cherry-picked this commit on top of extwitter 0.12.4. Not going to guarantee being up to date, but for now it's a solution. https://github.com/KristerV/extwitter

parroty commented 3 years ago

I'm sorry about not responding to this item. It might be too late, but if you have working change on top of the latest version, is it possible for you to open PR?

KristerV commented 3 years ago

Ideally you would rebase this one, at least GitLab has that option right in this UI.

Anyway - if the button isn't here I made the PR: #148 As far as I can tell I did preserve @pjskennedy's name in the commit

And I have this version in use at the moment - seems to work great.

pjskennedy commented 3 years ago

Hi @parroty and @KristerV

I have rebased, this should be good to merge now.

parroty commented 3 years ago

Thank you for the updates, and I'm sorry about taking so long to merge.

KristerV commented 3 years ago

hey no worries and thanks for the package!