Accessing jupyter notebooks through browsermob-proxy does not work

[alex4200]

To debug selenium tests I am trying to use a proxy to record all request/responses communications. In particular, I am using browsermob-proxy (see here) which is running e.g. on port 8080.

To recreate this error you have to have browsermobproxy running on port 8080 (checkout the bin package from here and start the binary with browsermob-proxy --port 8080 &), some unimportant ipynb notebook (just as example) and the following python test script:

import time
from browsermobproxy import Client
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Define the Client to the running browsermob-proxy
proxy = Client("localhost:8080", {})

# Create the firefox webdriver
profile = webdriver.FirefoxProfile()
selenium_proxy = proxy.selenium_proxy()
profile.set_proxy(selenium_proxy) # SETTING PROXY 
driver = webdriver.Firefox(firefox_profile=profile)

# Navigate to the jupyter main page, and then to the jupyter notebook
proxy.new_har("jupyter")
driver.get("http://localhost:8888/tree?token=d3c806ff34792029a67a4186b720d7ed43127b1e8c26259d")
what = (By.XPATH, "//a/span[contains(text(),'Untitled.ipynb')]")
elem = WebDriverWait(driver, 5).\
                    until(EC.element_to_be_clickable(what))
elem.click()

time.sleep(50)
print (proxy.har)
driver.quit()

which you can start using python script.py (or whatever you name it). Additional packages required and used:

Used packages:

• selenium 3.11.0 (pip install selenium==3.11.0)
• geckodriver 0.19.1 (from https://github.com/mozilla/geckodriver/releases/download/v$GECKODRIVER_VERSION/geckodriver-v0.19.1-linux64.tar.gz)
• browsermob-proxy 2.1.4 (from the binary zip as mentioned above)
• jupyter notebook 5.6.0

The code opens the main jupyter page and clicks on the Untitled notebook. If you do the steps by hand it works and the jupyter notebook loads fine. If it is done with the script using browsermob-proxy, you will get a connection error:

In what cases do I expect such a connection failed error? What is the server missing?

Also, in case of the connection issue, the notebook server prints the following additional lines:

[W 2018-08-20 14:17:28.283 305933 log:47] 400 GET /jupyterhub/user/305933/api/kernels/d84016fb-4d6e-44c7-b577-7de563a9b5b2/channels?session_id=47ADDE787AA14E8381C05FD8490A764C (128.178.97.113) 1371.30ms referer=None
[W 2018-08-20 14:17:29.361 305933 handlers:257] Replacing stale connection: d84016fb-4d6e-44c7-b577-7de563a9b5b2:47ADDE787AA14E8381C05FD8490A764C
[W 2018-08-20 14:17:51.406 305933 handlers:257] Replacing stale connection: d84016fb-4d6e-44c7-b577-7de563a9b5b2:47ADDE787AA14E8381C05FD8490A764C
[W 2018-08-20 14:18:15.412 305933 handlers:257] Replacing stale connection: d84016fb-4d6e-44c7-b577-7de563a9b5b2:47ADDE787AA14E8381C05FD8490A764C

which are not present if you run the test manually or without the proxy! So there must be something not working with the proxy, but what exactly...?

[alex4200]

Additional information:

In the browser console the following error is shown:

Firefox can’t establish a connection to the server at ws://localhost:8889/api/kernels/7e5ade90-3880-43c9-a383-9c11d32a5b70/channels?session_id=979ed84181164476b45c04a58b0ab195.

Maybe it has to do with websockets which are not supported by browsermobproxy?

[takluyver]

If browsermobproxy doesn't support websockets, Jupyter's not going to work with it.

[alex4200]

There is a solution (workaround) as described here: https://softwarequalityweb.wordpress.com/2018/09/13/monitoring-network-traffic-in-automated-web-tests/