Test Run Error: "Error configuring traffic-shaping"

[joacim-boive]

Environment: macOS Sierra 10.12.4 Linux Headless agent

Using latest wptagent and latest wptserver from repo.

Can I disable traffic shaping? It's not really necessary for my current purposes. modprobe doesn't exist on mac - the equivalent would be kextutil (I guess)

http://10.0.1.19:32771 "GET /work/getwork.php?f=json&shards=1&location=Local-WPTDriver&pc=Docker&version=170502.200558&screenwidth=1920&screenheight=1200&freedisk=1.117 HTTP/1.1" 200 216 10:59:49.080 - Job: {"processResults":1,"runs":2,"bwIn":0,"plr":0,"latency":0,"url":"https:\/\/www.lensway.se","Test ID":"170503_J5_2","Capture Video":1,"IPAddr":"172.17.0.1","bwOut":0,"orientation":"default","lighthouseTrace":1,"timeline":1,"timelineStackDepth":0,"browser":"Chrome"} 10:59:49.081 - [{'record': True, 'command': 'navigate', 'target': u'https://www.lensway.se'}] 10:59:49.093 - Preparing browser 10:59:49.093 - Terminating all instances of chrome chrome: no process found 10:59:49.098 - Flushing DNS dnsmasq: unrecognized service rndc: neither /etc/bind/rndc.conf nor /etc/bind/rndc.key was found 10:59:49.307 - Clearing profile /wptagent/work/Docker/browser.170503_J5_2.1 10:59:49.308 - /opt/google/chrome/chrome --disable-background-networking --no-default-browser-check --no-first-run --process-per-tab --new-window --disable-infobars --disable-translate --disable-notifications --disable-desktop-notifications --disable-save-password-bubble --allow-running-insecure-content --disable-component-update --disable-background-downloads --disable-add-to-shelf --disable-client-side-phishing-detection --disable-datasaver-prompt --disable-default-apps --disable-domain-reliability --safebrowsing-disable-auto-update --disable-background-timer-throttling --disable-sync --host-resolver-rules="MAP cache.pack.google.com 127.0.0.1","MAP clients1.google.com 127.0.0.1" --window-position="0,0" --window-size="1024,768" --remote-debugging-port=9222 --user-data-dir="/wptagent/work/Docker/browser.170503_J5_2.1" --disable-gpu --no-sandbox 10:59:49.335 - Starting new HTTP connection (1): localhost 10:59:49.336 - Connect to dev tools Error: HTTPConnectionPool(host='localhost', port=9222): Max retries exceeded with url: /json (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f0e513d3a10>: Failed to establish a new connection: [Errno 111] Connection refused',)) [819:824:0503/105949.448093:ERROR:bus.cc(427)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory 10:59:49.838 - Starting new HTTP connection (1): localhost 10:59:49.839 - http://localhost:9222 "GET /json HTTP/1.1" 200 374 10:59:49.840 - Dev Tools tabs: [{"description":"","title":"New Tab","url":"chrome:\/\/newtab\/","webSocketDebuggerUrl":"ws:\/\/localhost:9222\/devtools\/page\/f17c2297-71ff-4743-8746-a04d2eb82283","type":"page","id":"f17c2297-71ff-4743-8746-a04d2eb82283","devtoolsFrontendUrl":"\/devtools\/inspector.html?ws=localhost:9222\/devtools\/page\/f17c2297-71ff-4743-8746-a04d2eb82283"}] 10:59:49.841 - Connect to dev tools websocket Error: [Errno 111] Connection refused 10:59:49.842 - Devtools connected 10:59:49.842 - Sending: {"params":{},"id":1,"method":"Network.clearBrowserCache"} 10:59:49.842 - DevTools websocket connected 10:59:49.844 - {"id":1,"result":{}} 10:59:49.844 - Sending: {"params":{},"id":2,"method":"Network.clearBrowserCookies"} 10:59:49.846 - {"id":2,"result":{}} 10:59:49.846 - Sending: {"params":{"returnByValue":true,"expression":"navigator.userAgent"},"id":3,"method":"Runtime.evaluate"} 10:59:50.114 - {"id":3,"result":{"result":{"type":"string","value":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.96 Safari/537.36"}}} 10:59:50.114 - Sending: {"params":{"url":"about:blank"},"id":4,"method":"Page.navigate"} 10:59:50.282 - {"id":4,"result":{"frameId":"979.1"}} 10:59:50.282 - Waiting for Idle... 10:59:50.889 - Configuring traffic shaping: 0/0 - 0 ms, 0.00% plr 10:59:50.890 - sudo tc qdisc add dev ifb0 root handle 1:0 netem delay 0ms Cannot find device "ifb0" 10:59:50.899 - Resetting traffic shaping Cannot find device "ifb0" 10:59:50.908 - DevTools websocket disconnected

[pmeenan]

Yeah, shaping hasn't been implemented for OSX yet. It actually hasn't been tested on OSX at all so you may bump into a few issues here and there.

You disable shaping in locations.ini on the server. In the location for the test agent add:

connectivity=LAN

LAN can be anything, it's just a display string but when connectivity= is defined then shaping will be disabled.

[joacim-boive]

Hm, that didn't make any difference - except that I only can choose Connection: LAN.

Works running thru a Windows virtual machine using the same webpagetest server. But, I was hoping to avoid that and using Docker for both the server and the agent.

Any ideas would be appreciated. :)

[pmeenan]

try also adding --shaper none to the command-line for the agent.

[franjohn21]

I wrote up a post on my experience setting up webpagetest for mac os and turning off traffic shaping. Let me know if there's anything you have found for better workarounds.

[joacim-boive]

I did work around it by setting up a "normal" virtual machine as a client running Windows. But I'll try @franjohn21 solution - looks great from what I read.

Thanks!

[pmeenan]

FWIW, traffic-shaping is now fully supported on OSX (as of a few minutes ago) as is Firefox with Video capture. It should be feature-compatible with the Windows and Linux agents now.

[franjohn21]

That's awesome @pmeenan. Thanks

[gitjpuser]

Websocket Error: Network Error 12002, The operation timed out Due to this error message in console i am not execute my wpt script.

[pmeenan]

@gitjpuser Can you open a new issue and include the log from the console? That is a different issue from traffic-shaping.

[joacim-boive]

@pmeenan Is there any information available that you could point me too regarding setting up traffic shaping on OSX I'd appreciate it. ;)

[pmeenan]

If you are using the agent code from after the change was pushed it should just work (no command-line flags).

The one thing that needs to be in place is that the sudo access for the user running the agent needs to not prompt for password. That is done by running "sudo visudo" with an entry like this: https://askubuntu.com/a/574454

If you are interested in how to do it from the command-line for other purposes, the code is here

[franjohn21]

If we run it through the linux headless agent via Docker on an OS X host, traffic shaping still does not work in that case though, right? Or are there other configuration steps required?

[pmeenan]

Traffic-shaping docker containers is difficult at best generally but even more difficult when it's not on a Linux host. Good thread here.

It certainly won't work out of the box and would need some dev work if someone can figure out what magical incantations work (if it is even possible from within the container). Full virtualization is heavier but works a lot better.

[jpvincent]

For a docker container inside an OSX, trafic shaping still does not work. I also did not manage to have it work on an Ubuntu from within a windows (but it does work on a normal Ubuntu). The sitespeed.io team had the same problem and ended up with an utility that throttle the entire machine : https://github.com/sitespeedio/throttle If that works for you, I ended up with this script :

sudo throttle --up 6000 --down 12000 --rtt 70
docker run --cap-add=NET_ADMIN -p 4001:80 \
    --rm \
    --privileged \
    --network="host" \
    -e "SERVER_URL=http://localhost:3999/work/" \
    -e "LOCATION=Test" \
    -e "SHAPER=none" \
    webpagetest/agent:20.06
sudo throttle stop

it throttles the host, runs the agent with no trafic shaping, then you can make your test from your private instance (docker run -d -p 3999:80 --rm webpagetest/server, open localhost:3999 and remember to select "Native" in the Connection menu of the Advanced options). When your test is done, exit and throttling stops.