catchpoint / WebPageTest.agent

Cross-platform WebPageTest agent
Other
213 stars 138 forks source link

Added support for posting successful tests to pubsub #510

Closed pmeenan closed 2 years ago

pmeenan commented 2 years ago

This adds support for 2 new test job options:

pubsub_completed_queue - A pub/sub queue where successfully completed test jobs are posted to (failures need to have a retry queue configured) pubsub_completed_metrics - A list of metrics to include from the page data in the pubsub job (minimal set of just the metrics needed for the automation)

In the HA case, we pass the top-level URLs with some metadata indicating that they are top-level and configure them to write successful results to a pub/sub queue along with our custom metric crawl_links which is a priority-sorted list of same-origin links in the viewport.

The jobs then get resubmitted by a central controller that monitors the queues. It decides how many links to visit, generates new test ID's and submits the new actual tests (without a completion queue configured since we don't want to crawl another layer deeper).