search

sitespeedio / sitespeed.io

sitespeed.io is an open-source tool for comprehensive web performance analysis, enabling you to test, monitor, and optimize your website’s speed using real browsers in various environments.
https://www.sitespeed.io/
MIT License
4.76k stars 603 forks source link

Error generating url/summary/index #4124

Open ryanyango opened 8 months ago

ryanyango commented 8 months ago

Your question

Hi, Im getting this error at the end of my run when i generate the report:

`ERROR: Could not generate url/summary/index, /usr/src/app/lib/plugins/html/templates/url/thirdparty/index.pug:24 22| h2 Third party 23|

24| - const thirdPartyWebVersion = pageInfo.data.coach.run ? pageInfo.data.coach.run.thirdPartyWebVersion : pageInfo.data.coach.pageSummary.thirdPartyWebVersion; 25| p Third party requests categorised by 26| a(href='https://github.com/patrickhulce/third-party-web') Third party web 27| |  version #{thirdPartyWebVersion}.

Cannot read properties of undefined (reading 'run') [2024-03-28 00:36:43] ERROR: TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined at writeFile (node:fs:2241:5) at node:internal/util:442:7 at new Promise () at writeFile (node:internal/util:428:12) at write (file:///usr/src/app/lib/core/resultsStorage/storageManager.js:26:10) at file:///usr/src/app/lib/core/resultsStorage/storageManager.js:113:9 at async HTMLBuilder.render (file:///usr/src/app/lib/plugins/html/htmlBuilder.js:340:7) Errors while running: Rejected { "uuid": "64dd553c-2196-4a86-9831-dc607c4c639f", "type": "sitespeedio.render", "timestamp": "2024-03-28T00:36:41+00:00", "source": "queueHandler", "data": "{...}" } for plugin: html {"code":"ERR_INVALID_ARG_TYPE"} Error: Errors while running: Rejected { "uuid": "64dd553c-2196-4a86-9831-dc607c4c639f", "type": "sitespeedio.render", "timestamp": "2024-03-28T00:36:41+00:00", "source": "queueHandler", "data": "{...}" } for plugin: html {"code":"ERR_INVALID_ARG_TYPE"} at start (file:///usr/src/app/bin/sitespeed.js:178:15) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async file:///usr/src/app/bin/sitespeed.js:189:1`

What does the error generally indicate and how would i got about troubleshooting this? This occurs in the sitespeedio/sitespeed.io:33.4.0 and sitespeedio/sitespeed.io:latest images.

soulgalore commented 8 months ago

Hi @ryanyango thank you for creating the issue! Can you share how you run and what URL you test so I can recreate it?

Thanks Peter

ryanyango commented 8 months ago

Hi @soulgalore i run the following command:

docker run --rm -v "$(pwd):/sitespeed.io" sitespeedio/sitespeed.io:latest -b chrome --multi ssio_workflow.mjs -n 1 --video sharing the url wont help because its an internal test environment. What would be some steps i could take to start investigating the cause?

soulgalore commented 8 months ago

It looks like no coach data is collected from one page. I would watch the browser and check if the script do what's its supposed to do: https://www.sitespeed.io/documentation/sitespeed.io/scripting/tutorial-07-Debugging-Scripts.html#watch-what's-going-on - maybe there's one step that do not work?

yashgarde commented 4 months ago

I am also facing a similar issue:

Could not generate url/iteration/index, /Users/yash.garde/.nvm/versions/node/v20.9.0/lib/node_modules/sitespeed.io/lib/plugins/html/templates/url/cpu/index.pug:1
  > 1| - const browsertime = medianRun ? (pageInfo.data.browsertime ? pageInfo.data.browsertime.pageSummary.browserScripts[medianRun.runIndex - 1]: undefined) : pageInfo.data.browsertime.run 
    2| 
    3| - const cpu = medianRun ? (pageInfo.data.browsertime ? pageInfo.data.browsertime.pageSummary.cpu[medianRun.runIndex - 1]: undefined) : pageInfo.data.browsertime.run.cpu
    4| 

Cannot read properties of undefined (reading 'run')

This is also preceded by another interesting error message: [2024-07-09 14:33:58] DEBUG: [sitespeedio.testrunner.773e78df-15f4-4ecd-b6e0-7403525768f5] [2024-07-09 14:33:58] ERROR: [browsertime] Could not add fullyLoaded metric to URL undefined, we have statistic for the URLs

For context, I am using Onlinetest all installed locally for the time being as well as a local (non-Dockerized) sitespeed.io to run my tests. The test case is run for 1 iteration and when I observe the browser, it is correctly reaching the final page of the test case via a script I have written.

Also unable to share URL's as it is an internal test environment that we are testing against.

Any suggestions for this?

soulgalore commented 4 months ago

Hi @yashgarde cool, can you please share the script so I can reproduce it? It's really hard just to guess. Thank you!

yashgarde commented 4 months ago

Sorry for the delayed response, been trying to figure out the problem myself. I am unable to provide the exact script I am using that encounters the issue as it accesses internal webpages that we are targeting for performance testing. I will include the details of the nature of the tests and the details of what I have found so far below.

Here are some additional findings:

  1. The problem seems to occur only when using the --cpu=true configuration for sitespeed.io. With all other configs staying the same, adding the --cpu flag is what causes tests to fail with the following error: 
    
[2024-07-18 13:58:33] ERROR: [sitespeedio.testrunner.4bec25b1-2a96-404a-9a70-b072f4eb00e9] Could not run sitespeed.io ExecaError: Command failed with exit code 1: sitespeed.io --config /Users/yash.garde/default/4bec25b1-2a96-404a-9a70-b072f4eb00e9/default-4bec25b1-2a96-404a-9a70-b072f4eb00e9-config.json --storeResult /Users/yash.garde/default/4bec25b1-2a96-404a-9a70-b072f4eb00e9/default-4bec25b1-2a96-404a-9a70-b072f4eb00e9-result.json --disableAPI true /Users/yash.garde/default/4bec25b1-2a96-404a-9a70-b072f4eb00e9/Test_Click_Through.mjs '--viewPort=1280x1024' -n 1 --multi --slug grafana_test_cpu

[2024-07-18 13:58:32] ERROR: Could not add fullyLoaded metric to URL undefined, we have statistic for the URLs ["https://perftoolswptdedicated.service-now.com/now/nav/ui/classic/params/target/incident.do%3Fsys_id%3D57af7aec73d423002728660c4cf6a71c%26sysparm_record_target%3Dincident%26sysparm_record_row%3D1%26sysparm_record_rows%3D67%26sysparm_record_list%3DORDERBYDESCnumber"] [2024-07-18 13:58:33] ERROR: Could not generate url/iteration/index, /Users/yash.garde/.nvm/versions/node/v20.9.0/lib/node_modules/sitespeed.io/lib/plugins/html/templates/url/cpu/index.pug:1

1| - const browsertime = medianRun ? (pageInfo.data.browsertime ? pageInfo.data.browsertime.pageSummary.browserScripts[medianRun.runIndex - 1]: undefined) : pageInfo.data.browsertime.run 2| 3| - const cpu = medianRun ? (pageInfo.data.browsertime ? pageInfo.data.browsertime.pageSummary.cpu[medianRun.runIndex - 1]: undefined) : pageInfo.data.browsertime.run.cpu 4|

Cannot read properties of undefined (reading 'run') node:internal/errors:497 ErrorCaptureStackTrace(err); ^

TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined at writeFile (node:fs:2241:5) at node:internal/util:442:7 at new Promise () at writeFile (node:internal/util:428:12) at write (file:///Users/yash.garde/.nvm/versions/node/v20.9.0/lib/node_modules/sitespeed.io/lib/core/resultsStorage/storageManager.js:26:10) at file:///Users/yash.garde/.nvm/versions/node/v20.9.0/lib/node_modules/sitespeed.io/lib/core/resultsStorage/storageManager.js:113:9 { code: 'ERR_INVALID_ARG_TYPE' }

Node.js v20.9.0

2. Test scripts that involve simple navigations to pages and measuring performance are not impacted by this.
3. The scripts that are failing involve the use `commands.click.byJsAndWait` where the argument to the function is used to navigate through the shadow dom of the webpage to reach and click the required elements.

Interestingly, not all click through based tests fail due to the `--cpu` flag. The scripts that are failing currently have a final measurement step of click on a hyperlink that is within an embedded iFrame (all this is of course layered within the shadow dom of the webpage as well).
4. I have tried to replicate this issue with publicly available webpages by simulating similar click through user journeys but have struggled to find a similarly layered shadow dom (which I don't believe is the issue here) with embedded iFrames.

Hopefully that gives you an idea of the nature of the testing, I am providing a sample version of the script I am using (with the URLs redacted) to give you an idea as well:

export default async function (context, commands) { await commands.navigate('');

commands.meta.setTitle('Click first incident');
await commands.click.byJsAndWait('<redacted>');
await commands.click.byJsAndWait('<redacted>');

await commands.measure.start('Click first INC');
// Click on link within iFrame here
await commands.click.byJsAndWait('<redacted>');
return commands.measure.stop();

}



Additionally, when viewing the browser to see if the correct pages are loaded, all the right clicks and page loads are visible in the browser when running this setup locally.
soulgalore commented 4 months ago

Ok, hmm. Do you use Chrome or Firefox?

[2024-07-18 13:58:32] ERROR: Could not add fullyLoaded metric to URL undefined, we have statistic for the URLs

That means that in the HAR file, the URL for the page does not match the URL of the browser window. In this case it does not have a URL in the HAR. The HAR is generated differently depending if its Chrome or Firefox, could you try if it works in the other browser? Maybe I could add some better logging so it's easier to see what's going on. Is the HAR file created, then you could drag/drop it into http://www.softwareishard.com/har/viewer/ and see what it looks like (is a page missing etc).

If you do not add --cpu don't you get the same error? Like the HTML is generated but you see the error?

yashgarde commented 4 months ago

I am only testing on Chrome at the moment, I will have a look tomorrow and see whether the HAR file is generated in the error case. I will try it out with Firefox as well and get back to you here.

To answer your second question, without the --cpu flag and the same erroneous test, I do see the same error message but with a successfully generated HTML result and also the fullyLoaded metric is calculated and rendered correctly.

soulgalore commented 4 months ago

also the fullyLoaded metric is calculated and rendered correctly.

When I looked at the code, there's some logic to go through the HAR file and find the pages of each request and there something is broken because you get a URL that is undefined. Even though the HTML result is generated correctly, I think the HAR file is broken somehow. Can you share it or if you inspect it yourself? I would check for requests that do not match any page or a HAR that do not validate (drag/dropping the HAR into the HAR viewer will validate it).

soulgalore commented 4 months ago

Also another way to see what's wrong would to be to collect the trace log for Chrome (that file is used for creating the HAR file). If you add --browsertime.chrome.collectPerfLog true to your test, that file is stored for each URL. In your data folder for each URL you will see a file named chromePerflog-1.json.gz. Those events are parsed using https://github.com/sitespeedio/chrome-har to generate the HAR. If you can provide that file too, then I think I could backtrack the error.

yashgarde commented 4 months ago

Back with an update:

  1. I ran the same test script (all locally this time since I am experiencing the same issue regardless of environment) both with and without the --cpu flag. In both cases, the HAR file was successfully generated and both were valid when I put them in the website/tool that you provided earlier

  2. I ran it with the --browsertime.chrome.collectPerfLog true as you requested as well an do see the generateed file each time. I am attaching a compressed file with the HAR file and the chromePerflog files for both the with and without --cpu flag runs. sitespeed-data.zip

yashgarde commented 4 months ago

Another update: Just tried running the test with the --cpu flag on Firefox and the test completes but there is no CPU data populated in the CPU tab under Pages in the HTML view.

soulgalore commented 4 months ago

That seems to be a bug for Firefox, I created https://github.com/sitespeedio/sitespeed.io/issues/4233 for that.

soulgalore commented 4 months ago

Back with an update:

And the HAR files looks correct for you? By correct I mean all pages you test is included and requests looks ok? When I looked in the HARviewer the result looked identical for me but maybe I missed something? I will look a the perflog later.

soulgalore commented 4 months ago

[2024-07-18 13:58:32] ERROR: Could not add fullyLoaded metric to URL undefined, we have statistic for the URLs

Ok, there's a bug in the code, where I simplified getting a URL for each page in the HAR that do not always work. I'll fix that but it will take a day or two. The second page in the HAR, that is a page that you navigate to (https://XXX/api/now/session/notification)?

yashgarde commented 4 months ago

Sorry for the delayed response, the HAR files look good to me. I do not actually navigate to the URL you mentioned (/api/session/notification) so I'm not entirely sure why that is in the HAR. I'll have a look at that on my end and confirm if that is expected behavior or not.

Cheers for finding the bug, let me know if there's a way I can help at all.

yashgarde commented 4 months ago

Update: the /api/session/notification in the HAR is expected behavior even though it is NOT a page we I explicitly navigate to in the script. It is a request that is part of the page load.

Additionally, I noticed that there is a difference between the HAR file I pulled from the sitespeed-results directory on the Onlinetest testrunner and the one I receive when I click Download HAR on the results HTML. Is this expected behavior?

soulgalore commented 4 months ago

Ok, so that notification-URL is that in an iframe or can you see how that request is happening? I guess it can be a bug in chrome-har then, I haven't had time yet to check trace file yet, I hope to do that later this week.

soulgalore commented 4 months ago

I had a quick look today but I wasn't able to fix it. I need to check the documentation for the Chrome events, we have many tests but some of them are using a really old version of the events and when I've done my changes, the tests broke. But I wonder if it's not the Chrome implementation that changed over time and that I should recreate all the test cases. I'll hope to be able to work on it this weekend.

yashgarde commented 4 months ago

Don't know if this is related to this issue at all, but one of my user journey tests (where I am only measuring the final page load) is currently throwing this error:

[2024-07-29 15:40:42] ERROR: RangeError: Invalid time value
    at Date.toISOString (<anonymous>)
    at m.toISOString (/home/ubuntu/.nvm/versions/node/v20.9.0/lib/node_modules/sitespeed.io/node_modules/chrome-har/node_modules/dayjs/dayjs.min.js:1:6270)
    at module.exports (/home/ubuntu/.nvm/versions/node/v20.9.0/lib/node_modules/sitespeed.io/node_modules/chrome-har/lib/entryFromResponse.js:189:53)
    at harFromMessages (/home/ubuntu/.nvm/versions/node/v20.9.0/lib/node_modules/sitespeed.io/node_modules/chrome-har/index.js:410:15)
    at getHar (file:///home/ubuntu/.nvm/versions/node/v20.9.0/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/chrome/har.js:35:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Chromium.afterPageCompleteCheck (file:///home/ubuntu/.nvm/versions/node/v20.9.0/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/chrome/webdriver/chromium.js:288:9)
    at async Measure.stop (file:///home/ubuntu/.nvm/versions/node/v20.9.0/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/core/engine/command/measure.js:440:17)
    at async default (file:///home/ubuntu/default/4bcf1c7f-f8a7-43f5-91ce-8aa47103b163/Click_1st_INC.mjs:8:5)
    at async file:///home/ubuntu/.nvm/versions/node/v20.9.0/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/core/engine/run.js:4:7
    at async Iteration.run (file:///home/ubuntu/.nvm/versions/node/v20.9.0/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/core/engine/iteration.js:162:9)
    at async Engine.runByScript (file:///home/ubuntu/.nvm/versions/node/v20.9.0/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/core/engine/index.js:308:20)
    at async analyzeUrl (file:///home/ubuntu/.nvm/versions/node/v20.9.0/lib/node_modules/sitespeed.io/lib/plugins/browsertime/analyzer.js:202:12)
    at async BrowsertimePlugin.processMessage (file:///home/ubuntu/.nvm/versions/node/v20.9.0/lib/node_modules/sitespeed.io/lib/plugins/browsertime/index.js:173:26)

This is followed by the following error:

[2024-07-29 15:40:42] ERROR: No data to collect
[2024-07-29 15:40:42] ERROR: Could not add fullyLoaded metric to URL undefined, we have statistic for the URLs ["https://perftoolswptdedicated.service-now.com/now/nav/ui/classic/params/target/incident.do%3Fsys_id%3D57af7aec73d423002728660c4cf6a71c%26sysparm_record_target%3Dincident%26sysparm_record_row%3D1%26sysparm_record_rows%3D67%26sysparm_record_list%3DORDERBYDESCnumber","https://perftoolswptdedicated.service-now.com/now/nav/ui/classic/params/target/incident.do%3Fsys_id%3D57af7aec73d423002728660c4cf6a71c%26sysparm_record_target%3Dincident%26sysparm_record_row%3D1%26sysparm_record_rows%3D67%26sysparm_record_list%3DORDERBYDESCnumber?browsertime_run=2"]

which is an error we have seen before.

These tests were not run with the --cpu flag and the test script passes for a single iteration but fails on any more than 1.

Let me know if you need more info or if this is even related?

soulgalore commented 4 months ago

Hi @yashgarde can you please create another issue of that error. It looks like https://github.com/sitespeedio/chrome-har gets a timestamp from Chrome trace that is not correct. I can catch that but if you can share the trace I can make sure that its really work. Thanks!

yashgarde commented 4 months ago

Created a new issue #4243 as requested.

ryanyango-alt commented 2 months ago

Hi, is this issue still open?

I've had similiar findings below, my script also uses commands.click.byJsAndWait which seems to trigger theERROR: Could not add fullyLoaded metric to URL undefined, error:

Sorry for the delayed response, been trying to figure out the problem myself. I am unable to provide the exact script I am using that encounters the issue as it accesses internal webpages that we are targeting for performance testing. I will include the details of the nature of the tests and the details of what I have found so far below.

Here are some additional findings:

  1. The problem seems to occur only when using the --cpu=true configuration for sitespeed.io. With all other configs staying the same, adding the --cpu flag is what causes tests to fail with the following error:
[2024-07-18 13:58:33] ERROR: [sitespeedio.testrunner.4bec25b1-2a96-404a-9a70-b072f4eb00e9] Could not run sitespeed.io ExecaError: Command failed with exit code 1: sitespeed.io --config /Users/yash.garde/default/4bec25b1-2a96-404a-9a70-b072f4eb00e9/default-4bec25b1-2a96-404a-9a70-b072f4eb00e9-config.json --storeResult /Users/yash.garde/default/4bec25b1-2a96-404a-9a70-b072f4eb00e9/default-4bec25b1-2a96-404a-9a70-b072f4eb00e9-result.json --disableAPI true /Users/yash.garde/default/4bec25b1-2a96-404a-9a70-b072f4eb00e9/Test_Click_Through.mjs '--viewPort=1280x1024' -n 1 --multi --slug grafana_test_cpu

[2024-07-18 13:58:32] ERROR: Could not add fullyLoaded metric to URL undefined, we have statistic for the URLs ["https://perftoolswptdedicated.service-now.com/now/nav/ui/classic/params/target/incident.do%3Fsys_id%3D57af7aec73d423002728660c4cf6a71c%26sysparm_record_target%3Dincident%26sysparm_record_row%3D1%26sysparm_record_rows%3D67%26sysparm_record_list%3DORDERBYDESCnumber"] 
[2024-07-18 13:58:33] ERROR: Could not generate url/iteration/index, /Users/yash.garde/.nvm/versions/node/v20.9.0/lib/node_modules/sitespeed.io/lib/plugins/html/templates/url/cpu/index.pug:1
  > 1| - const browsertime = medianRun ? (pageInfo.data.browsertime ? pageInfo.data.browsertime.pageSummary.browserScripts[medianRun.runIndex - 1]: undefined) : pageInfo.data.browsertime.run 
    2| 
    3| - const cpu = medianRun ? (pageInfo.data.browsertime ? pageInfo.data.browsertime.pageSummary.cpu[medianRun.runIndex - 1]: undefined) : pageInfo.data.browsertime.run.cpu
    4| 

Cannot read properties of undefined (reading 'run')
node:internal/errors:497
    ErrorCaptureStackTrace(err);
    ^

TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined
    at writeFile (node:fs:2241:5)
    at node:internal/util:442:7
    at new Promise (<anonymous>)
    at writeFile (node:internal/util:428:12)
    at write (file:///Users/yash.garde/.nvm/versions/node/v20.9.0/lib/node_modules/sitespeed.io/lib/core/resultsStorage/storageManager.js:26:10)
    at file:///Users/yash.garde/.nvm/versions/node/v20.9.0/lib/node_modules/sitespeed.io/lib/core/resultsStorage/storageManager.js:113:9 {
  code: 'ERR_INVALID_ARG_TYPE'
}

Node.js v20.9.0
  1. Test scripts that involve simple navigations to pages and measuring performance are not impacted by this.
  2. The scripts that are failing involve the use commands.click.byJsAndWait where the argument to the function is used to navigate through the shadow dom of the webpage to reach and click the required elements.

Interestingly, not all click through based tests fail due to the --cpu flag. The scripts that are failing currently have a final measurement step of click on a hyperlink that is within an embedded iFrame (all this is of course layered within the shadow dom of the webpage as well). 4. I have tried to replicate this issue with publicly available webpages by simulating similar click through user journeys but have struggled to find a similarly layered shadow dom (which I don't believe is the issue here) with embedded iFrames.

Hopefully that gives you an idea of the nature of the testing, I am providing a sample version of the script I am using (with the URLs redacted) to give you an idea as well:

export default async function (context, commands) { 
    await commands.navigate('<redacted>');

    commands.meta.setTitle('Click first incident');
    await commands.click.byJsAndWait('<redacted>');
    await commands.click.byJsAndWait('<redacted>');

    await commands.measure.start('Click first INC');
    // Click on link within iFrame here
    await commands.click.byJsAndWait('<redacted>');
    return commands.measure.stop();
}

Additionally, when viewing the browser to see if the correct pages are loaded, all the right clicks and page loads are visible in the browser when running this setup locally.

soulgalore commented 2 months ago

@ryanyango-alt can you please share a example so I can test it and try to fix it? Thanks!

ryanyango-alt commented 2 months ago

@soulgalore

I cant give you all the details of the script, but the part that is erroring out is:

await commands.click.byJsAndWait('document.getElementById("cdv-tree").querySelectorAll("div[id*=ygtv] a:not([role=tab]):not(.summary-document)")[${i}]')

Note at this point of the script, its already at 3 nested iframes deep.

im running with the command:

docker run --rm -v "$(pwd):/sitespeed.io" sitespeedio/sitespeed.io:latest --multi my_url my_script.mjs -n 1 -b chrome