search

sitespeedio / sitespeed.io

sitespeed.io is an open-source tool for comprehensive web performance analysis, enabling you to test, monitor, and optimize your website’s speed using real browsers in various environments.
https://www.sitespeed.io/
MIT License
4.75k stars 603 forks source link

WebPageTest plugin doesn't work - code 404 #3771

Closed bgruszka closed 10 months ago

bgruszka commented 1 year ago

Have you read the documentation?

URL

https://www.onet.pl/

What are you trying to accomplish

I want to run sitespeed with WebPageTest plugin enabled, but unfortunately I have this: {"name":"WPTAPIError","code":404,"message":"Not Found"}

I set proper API key.

What browser did you use?

Other

How to reproduce

docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:26.1.0-webpagetest -b chrome --mobile -d 1 -n 1 -c 3g --webpagetest.key <REAL_KEY_HERE> --webpagetest.runs 1 --plugins.add /webpagetest https://www.onet.pl

Relevant log output

Chromium 97.0.4692.20 Ubuntu 20.04
Mozilla Firefox 105.0
[2023-02-27 11:55:50] INFO: Versions OS: linux 5.10.104-linuxkit nodejs: v16.16.0 sitespeed.io: 26.1.0 browsertime: 16.17.0 coach: 7.1.2
[2023-02-27 11:55:50] INFO: Sending url https://www.onet.pl to test on https://www.webpagetest.org
[2023-02-27 11:55:51] INFO: Running tests using Chrome - 1 iteration(s)
[2023-02-27 11:55:52] INFO: Testing url https://www.onet.pl iteration 1
[2023-02-27 11:56:03] ERROR: Could not run test for WebPageTest {"name":"WPTAPIError","code":404,"message":"Not Found"}
    at Object.<anonymous> (/webpagetest/node_modules/webpagetest/lib/helper.js:213:25)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/webpagetest/node_modules/webpagetest/lib/webpagetest.js:14:15)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/webpagetest/analyzer.js:5:21)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
[2023-02-27 11:56:04] INFO: Take after page complete check screenshot
[2023-02-27 11:56:05] INFO: Take cumulative layout shift screenshot
[2023-02-27 11:56:06] INFO: Take largest contentful paint screenshot
[2023-02-27 11:56:09] INFO: Get visual metrics from the video
[2023-02-27 11:56:15] INFO: https://www.onet.pl 205 requests, TTFB: 288ms, firstPaint: 424ms, firstVisualChange: 500ms, FCP: 424ms, DOMContentLoaded: 1.23s, LCP: 662ms, CLS: 0, TBT: 668ms, Load: 3.00s, speedIndex: 1.10s, visualComplete85: 2.04s, lastVisualChange: 4.50s
[2023-02-27 11:56:15] INFO: The server responded with a 401 status code for https://www.onet.pl/api/okonto/me?fields=userId%2Clogin%2Cav...
[2023-02-27 11:56:18] INFO: HTML stored in /sitespeed.io/sitespeed-result/www.onet.pl/2023-02-27-11-55-50
Vodrech commented 1 year ago

https://github.com/WebPageTest/webpagetest-api/pull/171/files Seems like they are deprecating sending in the webpagetest key as query parameters and replacing it by sending it in as a header, I feel this is already merged to the webpagetest as I also get 404, is this an easy fix? @soulgalore sorry for tag, but it has been like this for a while :)

soulgalore commented 1 year ago

I can help out, but what has changed and when? When I look in https://github.com/WPO-Foundation/webpagetest/tree/apache it looks like the last change was done three years ago. The https://github.com/WebPageTest/webpagetest-api seems to get changes all the time, but it's not versioned (like what version of the API works against which version of WebPageTest and more important what version works agains the Open Source version?

soulgalore commented 1 year ago

So then If I understand correctly they intentionally broke the API (by calling it "deprecated" but removing the functionality)?

Vodrech commented 1 year ago

Might have confused you a bit with above message, it's still possible to send in the api key via the query, I manually tested setting up the webpagetest plugin and running it in the docker with sitespeed.io image, tho I needed to use an older version because the documentation on sitespeed didn't work with setting up plugins manually.

The tests get passed into the webpagetest and I can see the result in there, but I think something is trowing in the try block in the analyzer.js, didn't have time to look further, as I don't have best practice with debugging in such an environment.

Vodrech commented 1 year ago

docker run --rm -v /Users/<user>/:/sitespeed.io sitespeedio/sitespeed.io:19.4.1 --plugins.add /sitespeed.io/test/plugin-webpagetest/ --webpagetest.location ec2-eu-north-1 --graphite.host <graphite-host> --webpagetest.browser chrome --webpagetest.connectivity 4G --webpagetest.lighthouse --webpagetest.runs 1 --webpagetest.key <api-key> --webpagetest.timeout 10000 --webpagetest.includeRepeatView --webpagetest.device iPhoneX --outputFolder <folder> --graphite.messages run --graphite.messages pageSummary --config /sitespeed.io/test/plugin-webpagetest/config.json /sitespeed.io/test/plugin-webpagetest/pages.txt

config.json:

    "plugins": {
        "remove": ["browsertime", "budget", "coach", "domains", "harstorer", "pagexray", "remove", "thirdparty", "tracestorer"]
    },
    "metrics": {
        "filter": [
              "webpagetest.pageSummary.data.lighthouse.categories.performance.score",
              "webpagetest.pageSummary.data.lighthouse.categories.accessibility.score",
              "webpagetest.pageSummary.data.lighthouse.audits.aria-allowed-attr.score",
              "webpagetest.pageSummary.data.lighthouse.audits.aria-required-attr.score",
              "webpagetest.pageSummary.data.lighthouse.audits.aria-required-children.score",
              "webpagetest.pageSummary.data.lighthouse.audits.aria-required-parent.score",
              "webpagetest.pageSummary.data.lighthouse.audits.aria-roles.score",
              "webpagetest.pageSummary.data.lighthouse.audits.aria-valid-attr-value.score",
              "webpagetest.pageSummary.data.lighthouse.audits.aria-valid-attr.score",
              "webpagetest.pageSummary.data.lighthouse.audits.errors-in-console.score",
              "webpagetest.pageSummary.data.lighthouse.audits.errors-in-console.numericValue",
              "webpagetest.pageSummary.data.median.firstView.chromeUserTiming_CumulativeLayoutShift"
        ]
    }

pages.txt:

bgruszka commented 1 year ago

@soulgalore @Vodrech Can I help you somehow?

soulgalore commented 1 year ago

@bgruszka what version of WebPageTest do you run against? I haven't looked yet what changed, I only looked in the API and I wanted to understand if that change killed the Apache branch or maybe I misunderstood. The webpagetest-api isn't versioned, and I think the problem is that if we upgrade it can work again their main branch (I don't remember what they call the main branch) but it will break the Apache branch integration?

Comeonautomaten commented 1 year ago

@soulgalore I have used sitespeed.io.26.1.0. It is over a month since it stopped working now (around Feb 8th, 2023), and hence losing metrics to graphana.

image: "sitespeedio/sitespeed.io:26.1.0-webpagetest"

Google Chrome 106.0.5249.119 Mozilla Firefox 106.0 Microsoft Edge 106.0.1370.52 [2023-02-27 05:45:25] INFO: Versions OS: linux 5.10.161+ nodejs: v16.16.0 sitespeed.io: 26.1.0 browsertime: 16.17.0 coach: 7.1.2

[2023-02-27 05:45:38] ERROR: Could not run test for WebPageTest {"name":"WPTAPIError","code":404,"message":"Not Found"}
    at Object.<anonymous> (/webpagetest/node_modules/webpagetest/lib/helper.js:213:25)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/webpagetest/node_modules/webpagetest/lib/webpagetest.js:14:15)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/webpagetest/analyzer.js:5:21)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
soulgalore commented 1 year ago

@Comeonautomaten what version are you using of WebPageTest? We didn't change anything in the API so I think it broke because of a change in WebPageTest?

Comeonautomaten commented 1 year ago

@soulgalore I do not know the exact version of WebPageTest. I used this image: sitespeedio/sitespeed.io:26.1.0-webpagetest as per documentation: https://www.sitespeed.io/documentation/sitespeed.io/webpagetest/#using-webpagetest This has been working for years now.

And, I see this merge request on WebPageTest: https://github.com/WebPageTest/webpagetest-api/pull/171

soulgalore commented 1 year ago

If I understand it correctly this is what has happened:

WebPageTest runs un-versionen and without a changelog in the master branch. The WebPageTest version got updated or autoupdated, the API doesn't work anymore since WebPageTest doesn't version handle the API against the WebPageTest server. The version of the WebPageTest API work against the Apache branch (or any of you using the Apache branch?). When that PR will be merged, it will work for WebPageTest API users that do tests against the master branch. Is that correct?

I don't want to update the WebPageTest API so we break compatibility against the Apache-branch?

soulgalore commented 1 year ago

Do you run private instances of WebPageTest? I tried to get an up and running using https://github.com/WPO-Foundation/webpagetest/tree/master/docker/local - I can make it run, but couldn't connect to the API, I always get the " Could not run test for WebPageTest {"statusCode":400,"statusText":"Invalid API key. To continue running tests via the WebPageTest API, you'll need to update your current key for the enhanced WebPageTest API. Read more here: https://product.webpagetest.org/api"" error.

Vodrech commented 1 year ago

Hello once again, I tried to follow the newly updated docs that you created, it was very beneficial, but somehow it seem like I'm doing something wrong.

I'm having trouble with adding the webpagetest plugin, I could not figure out if this actually was one of the 'default' plugins or not, but I tried both adding it via the cli and via config.json but neither did work, tried with the / slash as well and nothing changed, could you give me some guidance? @soulgalore

Image: sitespeedio/sitespeed.io:27.3.0-webpagetest Environment: Kubernetes

/start.sh --webpagetest.host <webpage_host> --webpagetest.location <location> --graphite.host <host> --webpagetest.key <secret> --webpagetest.connectivity 4G --webpagetest.lighthouse --webpagetest.mobile true --webpagetest.runs 1 --webpagetest.timeout 10000 --webpagetest.device iPhoneX --outputFolder <folder> --graphite.messages run --graphite.messages pageSummary --config <path>/config.json pages.txt

config.json

{
    "slug": "<slug>",
    "graphite": {
        "host": "<graphite_host>",
        "namespace": "sitespeed_io/webpagetest",
        "addSlugToKey": true
    },
    "plugins": {
        "list": true,
        "remove": ["browsertime"],
        "add": ["webpagetest"]
    },
    "metrics": {
        "filter": [
        <some_filters>
        ]
    }
}
Google Chrome 112.0.5615.49 
Mozilla Firefox 112.0
Microsoft Edge 112.0.1722.39 
[2023-04-18 15:48:56] INFO: Versions OS: linux 5.10.162+ nodejs: v18.14.2 sitespeed.io: 27.3.0 browsertime: 17.8.0 coach: 7.1.3
[2023-04-18 15:48:56] INFO: The following plugins are enabled: assets, budget, coach, domains, graphite, harstorer, html, lateststorer, metrics, pagexray, remove, text, thirdparty, tracestorer, webpagetest
Couldn't load plugin webpagetest: Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/usr/src/app/lib/plugins/webpagetest/index.js' imported from /usr/src/app/lib/core/pluginLoader.js
Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'webpagetest' imported from /usr/src/app/lib/core/pluginLoader.js
    at new NodeError (node:internal/errors:399:5)
    at packageResolve (node:internal/modules/esm/resolve:889:9)
    at moduleResolve (node:internal/modules/esm/resolve:938:20)
    at defaultResolve (node:internal/modules/esm/resolve:1153:11)
    at nextResolve (node:internal/modules/esm/loader:163:28)
    at ESMLoader.resolve (node:internal/modules/esm/loader:838:30)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:424:18)
    at ESMLoader.import (node:internal/modules/esm/loader:525:22)
    at importModuleDynamically (node:internal/modules/esm/translators:110:35)
    at importModuleDynamicallyCallback (node:internal/process/esm_loader:35:14) {
  code: 'ERR_MODULE_NOT_FOUND'
soulgalore commented 1 year ago

Hi @Vodrech hmm I think I missed updating the documentation for the plugin when sitespeed.io 27.0.0 was released. I fixed that now. If you use the prebuilt container the plugin path is /webpagetest/index.js

Comeonautomaten commented 1 year ago

Hi,

The issue seems to be fixed.

Google Chrome 112.0.5615.49 
Mozilla Firefox 112.0
Microsoft Edge 112.0.1722.39 
[2023-05-17 13:13:52] INFO: Versions OS: linux 5.10.162+ nodejs: v18.14.2 sitespeed.io: 27.3.0 browsertime: 17.8.0 coach: 7.1.3
[2023-05-17 13:13:52] INFO: The following plugins are enabled: assets, budget, coach, domains, graphite, harstorer, html, lateststorer, metrics, pagexray, remove, text, thirdparty, tracestorer, /webpagetest/index.js
[2023-05-17 13:13:53] INFO: Sending url https://www.xxxxxxx.com/xx to test on https://www.webpagetest.org/
soulgalore commented 1 year ago

Hi @Comeonautomaten thanks of sharing, it seems there was a bug then at the WebPageTest side.