brotab active/activate: urllib.error.HTTPError: HTTP Error 500: INTERNAL SERVER ERROR

[bjohas]

Hello,

brotab active

results in

Traceback (most recent call last):
  File "/home/_user/.local/bin/brotab", line 8, in <module>
    sys.exit(main())
  File "/home/_user/.local/lib/python3.8/site-packages/brotab/main.py", line 670, in main
    exit(run_commands(sys.argv[1:]))
  File "/home/_user/.local/lib/python3.8/site-packages/brotab/main.py", line 663, in run_commands
    result = args.func(args)
  File "/home/_user/.local/lib/python3.8/site-packages/brotab/main.py", line 175, in show_active_tabs
    tabs = api.get_active_tabs(args)
  File "/home/_user/.local/lib/python3.8/site-packages/brotab/api.py", line 115, in get_active_tabs
    return [self.prefix_tab(tab) for tab in self._get('/get_active_tabs').split(',')]
  File "/home/_user/.local/lib/python3.8/site-packages/brotab/api.py", line 249, in _get
    with urlopen(request, timeout=HTTP_TIMEOUT) as response:
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500: INTERNAL SERVER ERROR

I'm on Ubuntu 20.04, and this used to work (just ~2 days ago). Might be an update in Chrome?

(A similar issue occurs with brotab activate)

[balta2ar]

It's a known issue unfortunately. I think some of the commands make mediator fail and it stops responding in regular manner and doesn't recover...

[bjohas]

Hi @balta2ar,

Strangely, when I run the command via curl, it works. In other words, I cannot activate the tab via brotab, but I can activate with curl localhost.../...activate.... So the browser plugin works fine, it's just somehow the interaction with the brotab script that's broken.

[doctorcolossus]

@bjohas What CURL command were you using exactly?

I modified SingleMediatorsAPI._get() to output each URL it builds.

For me it queries the following URLs on brotab active:

$ brotab active
http://localhost:4625/get_pid
http://localhost:4625/get_browser
http://localhost:4626/get_pid
http://localhost:4626/get_browser
http://localhost:4627/get_pid
http://localhost:4627/get_browser
http://localhost:4625/get_active_tabs

The error occurs with the last URL (http://localhost:4625/get_active_tabs):

Traceback (most recent call last):
  File "/usr/bin/brotab", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.10/site-packages/brotab/main.py", line 670, in main
    exit(run_commands(sys.argv[1:]))
  File "/usr/lib/python3.10/site-packages/brotab/main.py", line 663, in run_commands
    result = args.func(args)
  File "/usr/lib/python3.10/site-packages/brotab/main.py", line 175, in show_active_tabs
    tabs = api.get_active_tabs(args)
  File "/usr/lib/python3.10/site-packages/brotab/api.py", line 115, in get_active_tabs
    return [self.prefix_tab(tab) for tab in self._get('/get_active_tabs').split(',')]
  File "/usr/lib/python3.10/site-packages/brotab/api.py", line 250, in _get
    with urlopen(request, timeout=HTTP_TIMEOUT) as response:
  File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.10/urllib/request.py", line 525, in open
    response = meth(req, response)
  File "/usr/lib/python3.10/urllib/request.py", line 634, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.10/urllib/request.py", line 563, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500: INTERNAL SERVER ERROR

And indeed, at least for me, CURLing that URL gives the same result:

$ curl "http://localhost:4625/get_active_tabs"
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>

Therefore I believe the problem does lie with the browser plugin, which I think is the mediator (sorry, new to the project and still learning the terminology), as @balta2ar said.

Similarly with brotab activate {id}:

http://localhost:4625/get_pid
http://localhost:4625/get_browser
http://localhost:4626/get_pid
http://localhost:4626/get_browser
http://localhost:4627/get_pid
http://localhost:4627/get_browser

http://localhost:4625/activate_tab/314
Traceback (most recent call last):
  File "/usr/bin/brotab", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3.10/site-packages/brotab/main.py", line 670, in main
    exit(run_commands(sys.argv[1:]))
  File "/usr/lib/python3.10/site-packages/brotab/main.py", line 663, in run_commands
    result = args.func(args)
  File "/usr/lib/python3.10/site-packages/brotab/main.py", line 168, in activate_tab
    api.activate_tab(args.tab_id, args.focused)
  File "/usr/lib/python3.10/site-packages/brotab/api.py", line 296, in activate_tab
    api.activate_tab(args, focused)
  File "/usr/lib/python3.10/site-packages/brotab/api.py", line 112, in activate_tab
    self._get('/activate_tab/%s%s' % (tab_id, '?focused=1' if focused else ''))
  File "/usr/lib/python3.10/site-packages/brotab/api.py", line 250, in _get
    with urlopen(request, timeout=HTTP_TIMEOUT) as response:
  File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.10/urllib/request.py", line 525, in open
    response = meth(req, response)
  File "/usr/lib/python3.10/urllib/request.py", line 634, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.10/urllib/request.py", line 563, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500: INTERNAL SERVER ERROR

$ curl http://localhost:4625/activate_tab/314
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>

I have little experience with browser plugins, so unfortunately it's a rabbithole I don't have time to descend at the moment. ☹️

[doctorcolossus]

In case this information is of any help or significance, I'm using Firefox 95.0.2 (64-bit) on Wayland using sway 1:1.6.1-2 on an up-to-date Arch Linux. I tried with brotab 1.3.0 from pip and have the same version of the browser plugin installed.

[bjohas]

Hi @doctorcolossus

Not sure whether this helps, but

> brotab active
...
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500: INTERNAL SERVER ERROR

while

> brotab clients
a.  localhost:4625  8702    <ERROR>
b.  localhost:4626  260701  chrome/chromium

and

> curl 'http://localhost:4625/get_active_tabs'
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>

and

> curl 'http://localhost:4626/get_active_tabs'
14.20

So it seems to me that there are 'stale' clients, and brotab active tries to query those, and chokes? Could the problem be fixed if brotab just ignored those errors, i.e. didn't catch the errors arising from 'stale' clients?

[doctorcolossus]

@bjohas Sorry - just saw this.

I think these are two interrelated issues.

1. As described by @balta2ar above:

   some of the commands make mediator fail and it stops responding in regular manner and doesn't recover...

2. as you noted, after the mediator fails:

   there are 'stale' clients, and brotab active tries to query those, and chokes

The second problem causes all subsequent brotab commands to fail, and I think your idea of not querying "stale" mediators would fix that. FWIW, my own lazy workaround for that for now is killall bt_mediator, then reloading the browser extension.

However, the question remains about what is causing the mediators to crash, how to avoid that, and how mediators can be changed to be more resilient and recover from other unexpected errors. So I believe there are at least three tasks:

1. identify and fix the bug(s) causing the mediator(s) to crash
2. modify the mediator(s) to recover from errors
3. fix the CLI so it doesn't query "stale" mediators

I can probably implement the third one, which would be in Python, but the first two tasks (involving browser plugins) are beyond my abilities for now.

[bjohas]

Hi @doctorcolossus, a fix for 3 would be amazing!

[doctorcolossus]

@bjohas Sorry, I am sometimes bad about keeping up with my github notifications and just saw this. I am overwhelmed with other tasks at the moment, but will put this on my to-do list and link to a pull-request here when I am able to get this done!

[bjohas]

Hi @doctorcolossus - not to worry! It's all very much appreciated!

[balta2ar]

Communication of mediator with browser has been subject to changes in the newly released 1.4.0, so I hope it works better now -- so give it a try. If anything, please comment here and I'll reopen the issue.

[bjohas]

Thank you!