search

magicalraccoon / tootstream

A command line interface for interacting with Mastodon instances
MIT License
259 stars 32 forks source link

Crash while streaming #163

Closed deejoe closed 6 years ago

deejoe commented 6 years ago

tootstream-crash-20180116a.txt

craigmaloney commented 6 years ago

Does this happen all the time or is this only on occasion? I tried streaming with octodon.social and didn't experience any issues.

deejoe commented 6 years ago

I have only restarted it once since capturing the stack trace shared recently.

Again, I saw only one toot come through before it abended again.

My environment for this instance is 64 bit Ubuntu 16.04 pretty much stock, mostly up to date on patches, running inside tmux.

craigmaloney commented 6 years ago

This appears to be an issue with mstdn.io:

(Pdb++) event
{'event': 'update\ndata: {"id":"99380245456482302",

The json is malformed so it isn't getting the data field.

On octodon.social it looks like the following:

{'data': '{"id":"99380338221564093"

and there are keys for event and data.

I think you need to speak with your administrator re: fixing the streaming JSON.

That said I'll take the opportunity to fix up the streaming so it doesn't crash the program when something goes awry.

Thanks!

craigmaloney commented 6 years ago

https://github.com/magicalraccoon/tootstream/commit/790d216d3a4ca9f6dc6b53c06db624766b5baa33 fixes this so it will catch the exception and show something useful instead of crashing outright.

Please check with this version and let us know how the conversation with the admin goes. It looks like something is seriously wrong with the streaming on that instance.

Closing the issue. Thanks!

angristan commented 6 years ago

This appears to be an issue with mstdn.io:

(Pdb++) event {'event': 'update\ndata: {"id":"99380245456482302",

Do you get this data from the API? How can I reproduce this?

craigmaloney commented 6 years ago

This is coming from Tootstream when it crashes.

How I run it to get to that point:

source /path/to/virtualenv/bin/activate
(tootstream)craig@lister:~$ python -m pdb `which tootstream` -P mstdn.io
[2] > /home/craig/.virtualenvs/tootstream/bin/tootstream(3)<module>()
-> __requires__ = 'tootstream==0.3.2'
(Pdb++) c
Tootstream version: 0.3.2
You are connected to https://mstdn.io
Enter a command. Use 'help' for a list of commands.

[@craigmaloney (mstdn.io)]:

And then...

stream fed or stream local or stream home

When it crashes:

p event

That should give you what it has collected up to that point.

(I use the debugger pdb++ which you can install in your virtualenv using pip install pdbpp. Install it before running the above python -m pdb command. pdb++ gives syntax higlihgiting and a more pleasant debugging experience.)

Hope this helps. LMK if you need some additional help with getting set up.

craigmaloney commented 6 years ago

If you're using the development branch this error will be trapped so use the 0.3.2 release for now to get the old behavior.

angristan commented 6 years ago

Ok I think I can reproduce:

Traceback (most recent call last):
  File "/root/tootstream/lib/python3.4/site-packages/Mastodon.py-1.2.1-py3.4.egg/mastodon/streaming.py", line 81, in _dispatch
    data = event['data']
KeyError: 'data'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.4/pdb.py", line 1661, in main
    pdb._runscript(mainpyfile)
  File "/usr/lib/python3.4/pdb.py", line 1542, in _runscript
    self.run(statement)
  File "/usr/lib/python3.4/bdb.py", line 431, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/root/tootstream/bin/tootstream", line 3, in <module>
    __requires__ = 'tootstream==0.3.2'
  File "/root/tootstream/lib/python3.4/site-packages/click-6.7-py3.4.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/root/tootstream/lib/python3.4/site-packages/click-6.7-py3.4.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/root/tootstream/lib/python3.4/site-packages/click-6.7-py3.4.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/root/tootstream/lib/python3.4/site-packages/click-6.7-py3.4.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/root/tootstream/lib/python3.4/site-packages/tootstream-0.3.2-py3.4.egg/tootstream/toot.py", line 1665, in main
    cmd_func(mastodon, rest)
  File "/root/tootstream/lib/python3.4/site-packages/tootstream-0.3.2-py3.4.egg/tootstream/toot.py", line 1052, in stream
    mastodon.stream_public(toot_listener)
  File "<decorator-gen-64>", line 2, in stream_public
  File "/root/tootstream/lib/python3.4/site-packages/Mastodon.py-1.2.1-py3.4.egg/mastodon/Mastodon.py", line 57, in wrapper
    return function(self, *args, **kwargs)
  File "/root/tootstream/lib/python3.4/site-packages/Mastodon.py-1.2.1-py3.4.egg/mastodon/Mastodon.py", line 1373, in stream_public
    return self.__stream('/api/v1/streaming/public', listener, async=async)
  File "/root/tootstream/lib/python3.4/site-packages/Mastodon.py-1.2.1-py3.4.egg/mastodon/Mastodon.py", line 1702, in __stream
    listener.handle_stream(r)
  File "/root/tootstream/lib/python3.4/site-packages/Mastodon.py-1.2.1-py3.4.egg/mastodon/streaming.py", line 49, in handle_stream
    self.handle_line(line_buffer)
  File "/root/tootstream/lib/python3.4/site-packages/Mastodon.py-1.2.1-py3.4.egg/mastodon/streaming.py", line 67, in handle_line
    self._dispatch(self.event)
  File "/root/tootstream/lib/python3.4/site-packages/Mastodon.py-1.2.1-py3.4.egg/mastodon/streaming.py", line 86, in _dispatch
    err
  File "<string>", line 3, in raise_from
mastodon.Mastodon.MastodonMalformedEventError: ('Missing field', 'data', {'event': 'update\ndata: {"id":"99457173906463110","created_at":"2018-02-02T18:27:17.398Z","in_reply_to_id":null,"in_reply_to_account_id":null,"sensitive":false,"spoiler_text":"","visibility":"public","language":"ja","uri":"https://mstdn.jp/users/BIOTONIC/statuses/99457173807057647","content":"<p>いや血はいるだろ</p>","url":"https://mstdn.jp/@BIOTONIC/99457173807057647","reblogs_count":0,"favourites_count":0,"reblog":null,"application":null,"account":{"id":"23964","username":"BIOTONIC","acct":"BIOTONIC@mstdn.jp","display_name":"幼女","locked":false,"created_at":"2017-04-16T22:37:28.791Z","note":"<p>EDM好きの幼女<br>RocketLeagueとアズレン</p>","url":"https://mstdn.jp/@BIOTONIC","avatar":"https://mstdn.io/system/accounts/avatars/000/023/964/original/9eaa5b974250a932.jpeg","avatar_static":"https://mstdn.io/system/accounts/avatars/000/023/964/original/9eaa5b974250a932.jpeg","header":"https://mstdn.io/system/accounts/headers/000/023/964/original/f4287b7a4c645f1b.jpeg","header_static":"https://mstdn.io/system/accounts/headers/000/023/964/original/f4287b7a4c645f1b.jpeg","followers_count":596,"following_count":67,"statuses_count":20167},"media_attachments":[],"mentions":[],"tags":[],"emojis":[]}\n\nevent: delete\ndata: 727899'})
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> <string>(3)raise_from()

However:

(Pdb) p event
*** NameError: name 'event' is not defined
craigmaloney commented 6 years ago

Right. It got out of sync when it was streaming the delete event (see the traceback near the end).

angristan commented 6 years ago

So if I understand correctly,

mastodon.Mastodon.MastodonMalformedEventError: ('Missing field', 'data', {'event': 'update\ndata: {"id":"99457173906463110","created_at":"2018-02-02T18:27:17.398Z","in_reply_to_id":null,"in_reply_to_account_id":null,"sensitive":false,"spoiler_text":"","visibility":"public","language":"ja","uri":"https://mstdn.jp/users/BIOTONIC/statuses/99457173807057647","content":"<p>いや血はいるだろ</p>","url":"https://mstdn.jp/@BIOTONIC/99457173807057647","reblogs_count":0,"favourites_count":0,"reblog":null,"application":null,"account":{"id":"23964","username":"BIOTONIC","acct":"BIOTONIC@mstdn.jp","display_name":"幼女","locked":false,"created_at":"2017-04-16T22:37:28.791Z","note":"<p>EDM好きの幼女<br>RocketLeagueとアズレン</p>","url":"https://mstdn.jp/@BIOTONIC","avatar":"https://mstdn.io/system/accounts/avatars/000/023/964/original/9eaa5b974250a932.jpeg","avatar_static":"https://mstdn.io/system/accounts/avatars/000/023/964/original/9eaa5b974250a932.jpeg","header":"https://mstdn.io/system/accounts/headers/000/023/964/original/f4287b7a4c645f1b.jpeg","header_static":"https://mstdn.io/system/accounts/headers/000/023/964/original/f4287b7a4c645f1b.jpeg","followers_count":596,"following_count":67,"statuses_count":20167},"media_attachments":[],"mentions":[],"tags":[],"emojis":[]}\n\nevent: delete\ndata: 727899'})

is the issue, and more specifically, \n\nevent: delete\ndata: 727899'?

craigmaloney commented 6 years ago

Correct. There's something getting weird in how the streaming piece that is causing it to not get delimited properly.

angristan commented 6 years ago

Wow that's super weird. I don't know what to do and where to search, but thanks for your help.

deejoe commented 6 years ago

https://gitlab.com/deejoe/bugs/blob/master/tootstream-mastotech.md

deejoe commented 6 years ago

Will mention the other bug if/when I get around to submitting it.

deejoe commented 6 years ago

@Angristan I never got around to asking you this before, maybe I should know this already, but do you have a repo up for mstdn.io configuration files, issue tracking, etc?

angristan commented 6 years ago

@deejoe No I have not