search

catchpoint / WebPageTest.agent

Cross-platform WebPageTest agent
Other
213 stars 138 forks source link

netlog functionality breaks agents on Ubuntu <20 #532

Closed rposbo closed 2 years ago

rposbo commented 2 years ago

The new netlog.py functionality is the only file in the repo to explicitly require python3: https://github.com/WPO-Foundation/wptagent/blob/master/internal/support/netlog.py

However, python3 is only installed for Ubuntu versions 20 and above: https://github.com/WPO-Foundation/wptagent/blob/eb3c3399224af524e0d722c485ac3c668feb27a8/ubuntu_install.sh#L11

There are LTS versions of Ubuntu going back to 14.04: https://wiki.ubuntu.com/Releases

Given that the netlog functionality is a new enhancement, and not a core change, can we make it fail gracefully instead of break older Ubuntu agents? (A bit more info: https://twitter.com/rposbo/status/1536314937504632837)

I'm not sure how we'd do this in python (not my main language)? Maybe bring the imports within netlog.py into a huge try/catch? Check for python3 before attempting to import netlog.py into the browser.py files? Have a try/catch around problematic sections of netlog.py, e.g.

try:
    from urllib.parse import urlparse
except ImportError:
     from urlparse import urlparse

I appreciate we have mitigations (disable polling for updates in our agents, regularly rebuild agents, etc), but this can help in setups where they're just left alone and prefer to pull updates down when available.

pmeenan commented 2 years ago

Sorry, ubuntu_install.sh should probably be nuked (I'll at least update it). Here is the supported installer script for generating an agent install. It is what was used for the old AMI's as well as for the production WPT agents and the HTTP Archive agents (all of which are on 18.04).

I wouldn't recommend 20.04 without a lot of testing. Last time I tried, Chrome's rendering performance was a fair bit slower than on 18.04 and all of the metrics regressed.

rposbo commented 2 years ago

Ooooh, I see! Thanks, closing the issue.

pmeenan commented 2 years ago

FWIW, I just updated ubuntu-install to match wptagent-install and added a prompt to the beginning of it to direct people to it for dedicated agent installs (technically it also supports non-dedicated installs too).

At some point we should nuke the installers from this repo entirely but it's currently the only place with centos automation.

rposbo commented 2 years ago

I'm thinking it's still worth wrapping this breaking change in something more defensive, so it fails silently on older agents.

What do you think? Should I re-open?

pmeenan commented 2 years ago

Python 2.7 was EoL over 2 years ago and is being removed in a bunch of places. The agent is web-exposed so not pulling in security fixes could be an issue.

Sorry about the jolt, we had assumed most users had rebuilt with wptagent-install once the public AMI's were shut down.

It's a rocky transition but getting any remaining private instance users onto 3.x is well worth it.