Open mrchrisadams opened 1 year ago
I vaguely remember some of this now, and profiler docs are helpful too.
The output that Firefox Profiler needs is generated by the Gecko Profiler, a piece of c++ code inside Firefox.
If you have the ability to run javascript in the context of driving the browser, there's a profiler
object you can interact with.
Here's the code from the corresponding docs profile page:
(() => {
// I think these are the settings telling the profile what to profile, how often to sample, and
// setting a limit of how many samples to create.
const settings = {
entries: 1000000,
interval: 0.4,
features: ["js", "stackwalk", "threads", "leaf"],
threads: ["GeckoMain", "Compositor"]
};
// start the profiler. and recording stuff
Services.profiler.StartProfiler(
settings.entries,
settings.interval,
settings.features,
settings.features.length,
settings.threads,
settings.threads.length
);
// dump the output somewhere, currently to the console I guess?
setTimeout(() => {
Services.profiler.getProfileDataAsync().then(profile => {
for (let i = 0; i < profile.threads.length; i++) {
const thread = profile.threads[i];
}
console.log(profile);
// Stopping the profiler will delete the data in the buffer.
Services.profiler.StopProfiler();
});
}, 500);
})();
I can't remember if sitespeed lets you run javascript in this context easily, or how best to write this to a file, but presumably, if you know how to do these two things, then it's not a significant leap to be able to consume them in the Profiler, or even expose some metrics for presentation inside sitespeed.
I think Browsertime might be able to do this, which Sitespeedio uses. If you can use Selenium to access the browser console as part of a test run, then I think you might be able to execute the code above using Browsertime, based on what Browsertime already lets you do:
Browsertime uses Selenium NodeJS to drive the browser. It starts the browser, load a URL, executes configurable Javascripts to collect metrics, collect a HAR file.
To get the HAR from Firefox we use the HAR Export Trigger and Chrome we use Chrome-HAR to parse the timeline log and generate the HAR file.
More below:
https://www.sitespeed.io/documentation/browsertime/introduction/
I don't know what overhead any of this would add on top of profiling. Still, it makes me think this is doable.
Hi @mrchrisadams we support getting the profile using --browsertime.firefox.geckoProfiler
. If you want to run it exactly as in the example it would look like:
sitespeed.io -b firefox --browsertime.firefox.geckoProfiler --browsertime.firefox.geckoProfilerParams.features "js, stackwalk, threads, leaf" --browsertime.firefox.geckoProfilerParams.thread "GeckoMain, Compositor" https://www.sitespeed.io -n 1
That will generate a geckoProfile-1.json.gz file, the problem is I don't fully understand the internal Firefox format, how do I see the energy consumption? I could build something parses the file but I need some documentation/hints how to get the juicy bits out of the file.
Oh sweet, I had no idea this was already supported Peter! I'm less familiar with the internal Firefox Format, but this issue here seems to be the one we'd want to watch, or ask questions about in meantime:
I've asked Florian at Mozilla about this - he's the author of that issue and has been the person I've been corresponding with.
Ok, so what this would allow you to do is set the specifics of the Firefox browser that you are using to test with sitespeed.io by specifically configuring it with something like --browsertime.firefox.geckoProfiler averageUser.json
or something like that.
Then when you're crawling a site, you can use the specifications of that Firefox browser, rather than whatever the default is when running Firefox as the rendering engine.
So something like this would work if you wanted to use that Firefox profile to evaluate 50 web pages with the SWD sustainability model:
docker run --rm -v "$(pwd):/sitespeed.io" sitespeedio/sitespeed.io:29.3.0 https://example.com -d 3 -m 50 -n 2 --sustainable.enable --axe.enable --details --description --browsertime.videoParams.filmstripQuality 5 --sustainable.model swd --firefox --browsertime.firefox.geckoProfiler averageUser.json
Here's a documentation to the profile file format: https://github.com/firefox-devtools/profiler/blob/main/src/types/profile.js and some some more documentation here: https://github.com/firefox-devtools/profiler/tree/main/docs-developer
Hey folks, we had an in-person chat with @mrchrisadams 2 days ago and I was made aware of the issue where browsertime fails when the "power" feature is set to capture power measurements.
After spending some time on it, I managed to reproduce and find the cause of issue. It looks like it's because of the geckodriver build, we both had the arm64 macbooks, and looks like the sitespeedio/geckodriver
library uses the x86_64 binaries all the time.
After https://github.com/sitespeedio/geckodriver/pull/14 this issue should be fixed. With this PR applied, I can successfully power profile with the sitespeed.io tool.
Here's the complete command I used, it's the exact replication of our "power" preset inside Firefox:
sitespeed.io -b firefox --browsertime.firefox.geckoProfiler --browsertime.firefox.geckoProfilerParams.bufferSize 134217728 --browsertime.firefox.geckoProfilerParams.interval 10 --browsertime.firefox.geckoProfilerParams.features "screenshots,js,stackwalk,cpu,processcpu,nostacksampling,ipcmessages,markersallthreads,power" --browsertime.firefox.geckoProfilerParams.threads "GeckoMain,Renderer" https://www.sitespeed.io -n 1
Here's an example profile output I got with this command: https://share.firefox.dev/3MWPGXZ
Note that this command will still fail on arm64 macOS machines until we land the sitespeedio/geckodriver
PR and publish a new version (and start using that version in both browsertime and sitespeed.io tools).
On the otherhand, I agree that we could make this easier by having these sensible defaults for power profiling in a json file like power.json
.
Oh sweet, thanks @canova!
Let me check if I understand.
We might define the params as a config file, power.json
, like so:
{
"browsertime": {
"browser": "firefox",
"iterations": 1,
"firefox": {
"geckoProfilerParams": {
"bufferSize": 134217728,
"interval": 10,
"features": "screenshots,js,stackwalk,cpu,processcpu,nostacksampling,ipcmessages,markersallthreads,power",
"threads": "GeckoMain,Renderer"
},
}
}
}
Which would mean we could then run sitespeed like this:
sitespeed.io https://example.org --config power.json
I think with the PRs mentioned above merged in, we'd have a way profile visiting a page, in an automated fashion check for regressions, and ideally get direct power measurements, rather than relying on modelled figures in SWD for end user devices.
Presumably, once we have that, it's then possible to run more compicated user journeys too. They're documented in more detail on the page below:
https://www.sitespeed.io/documentation/sitespeed.io/scripting/#measure-one-page-after-you-logged-in
For the purposes of this convo tho, I think you could have a file called login.js
containing the code below:
export default async function (context, commands) {
await commands.navigate(
'https://example.org'
);
try {
// Find the sign in button and click it
await commands.click.byId('sign_in_button');
// Wait some time for the page to open a new login frame
await commands.wait.byTime(2000);
// Switch to the login frame
await commands.switch.toFrame('loginFrame');
// Find the username fields by xpath (just as an example)
await commands.addText.byXpath(
'peter@example.org',
'//*[@id="userName"]'
);
// Click on the next button
await commands.click.byId('verifyUserButton');
// Wait for the GUI to display the password field so we can select it
await commands.wait.byTime(2000);
// Wait for the actual password field
await commands.wait.byId('password', 5000);
// Fill in the password
await commands.addText.byId('dejh8Ghgs6ga(1217)', 'password');
// Click the submit button
await commands.click.byId('btnSubmit');
// In your implementation it is probably better to wait for an id
await commands.wait.byTime(5000);
// Measure the next page as a logged in user
return commands.measure.start(
'https://example.org/logged/in/page'
);
} catch(e) {
// We try/catch so we will catch if the the input fields can't be found
// We could have an alternative flow ...
// else we can just let it cascade since it caught later on and reported in
// the HTML
throw e;
}
};
And then run the commmand, including those steps in an automated fashion, as well as the power.json
config:
sitespeed.io https://example.org --config power.json --preScript login.js
Is this how you'd see it too?
Where I need help understanding all this so we could close the issue
The main thing I think I am missing is how to convert the --browsertime.firefox.geckoProfiler
flag to the equivalent for placing into power.json
. That's something I think @soulgalore might have some insight on.
Once we have that, I think I'd understand this issue well enough to have a go at contributing it to the sitespeed docs, once the necessary PRs are merged in.
@mgifford this might interest you too :)
@canova so what I meant (or what I need), is that I need to understand the correct way to calculate the CPU usage? Summarising all threadCPUDelta for a profile? And then the unit differs per platform? Thinking the easiest way for getting usage, would be that we iterate over the geckoprofile output and generate one new metric, I can fix that as long someone help me with the correct way to calculate it :)
@mrchrisadams
Oh, I wasn't aware of the --config
option already. Yes, this config json file makes things a lot easier with the way you use it.
For --browsertime.firefox.geckoProfiler
, I think you should be able to do something like:
{
"browsertime": {
"firefox": {
"geckoProfiler": true,
...
}
...
}
}
(with the other options omitted)
@soulgalore
Ah I see. This threadCPUDelta
values show you the CPU usage and not the power usage values. They mostly correlate to each other but not exactly the same. But here's a good thing, you don't really need to extract CPU usage value because browsertime
already has CPU usage information in ms unit here:
https://github.com/sitespeedio/browsertime/blob/4547991bf0ebed6692f01ad7c94997652804aa1d/lib/firefox/webdriver/firefox.js#L179-L189
It doesn't need to have the profiler active, so it will be present all the time. I don't know if you display this already. We use this metric to track cpu usage in Firefox perf testing (the variable is called powerusage
here, which is a bit misleading though).
If you want to extract the power usage values, I think it's possible to extract a single energy usage value out of the profile data (but not as easy as the CPU usage). Let me try to explain how to extract this value of a profile json.
profile.meta.configuration.features
array and see if it contains "power" feature in the array. If it doesn't contain this feature it means that the power values are not recorded, so you can bail early.profile.counters
. But there are also other counters for other purposes. So iterate over these counters and find the ones with category: "power"
.Hope this helps! Also thanks for publishing all the packages with https://github.com/sitespeedio/geckodriver/pull/14 . @mrchrisadams you should be able to capture power profiles with the sitespeed.io and browsertime tools now with the latest version.
Thanks @canova that is super helpful! I started this morning to implement it, I'm almost done but I don't fully understand the counters data structure. What do the two numbers in each data array represent? Which one should I summarise?
{
name: 'Power: DRAM',
category: 'power',
description: 'RAPL DRAM',
sample_groups: [ { id: 0, samples: { schema: { time: 0, count: 1 },
data: [
[ 3759.491881, 777833777533 ],
[ 3770.838823, 3017850 ],
[ 3779.618929, 542534 ],
[ 3790.957486, 2577040 ],
[ 3800.470425, 2475315 ],
... } ]
}
@soulgalore Nice!
We use schema
to identify which data field corresponds to which value. If you look at the line above, you'll see this:
schema: { time: 0, count: 1 }
It means that first number is time
and the second number is count
. And count
is the one you're looking for. But instead of hardcoding 1
directly, I would recommend getting that from schema, like:
const countIndex = samples.schema.count;
for (const datum of samples.data) {
const count = datum[countIndex];
<...accumulate...>
}
This way it will be more future-proof in case of any change we make in the json format.
Also, it's good to mention that this counter object is changing slightly in Firefox 122 (see bug 1866629, currently in beta, will ship to release on January 23) We are removing sample_groups
array and keeping only one samples
object instead. Previously that sample_groups
were implemente thinking that we might need it in the future and it only contained a single samples
object. This bug I mentioned removes it since we never really needed it so far. See the new object type in here too.
To be able to handle these differences, we use version numbers in our json format. You can find that number in profile.meta.version
. We also have a changelog here. So if it's version >= 29, it won't have sample_groups
and and if it's version < 29 it will have it. Sorry for this annoyance, we usually keep the json format stable but make some changes rarely.
Thank you @canova ! Ok, that works fine. What do we call this metric? We have "powerUsage" today from Android phones (that's info that we get from dumpsys batterystats). I need fix so the metric is reported from browsertime. I'm thinking reporting the raw usage and then in sitespeed.io we can use co2 to convert it.
Hm, I'm not so sure about the name. We simply call them "power" in the Firefox Profiler but already having "powerUsage" makes it trickier.
I pushed a PR in Browsertime calling it powerConsumption, let me know if anyone wants to change the implementation or naming :)
When that rolls out, I can add it to sitespeed.io so we use CO2 to convert the Wh.
Ok, it will look like this:
browsertime --config power.json https://www.sitespeed.io -n 1
I added a flag that needs to be set including the "power" in features --firefox.powerConsumption
.
{
"browser": "firefox",
"iterations": 1,
"firefox": {
"powerConsumption": true,
"geckoProfiler": true,
"geckoProfilerParams": {
"features": "power"
}
}
}
Your question
Hi folks.
I think I asked this in slack aaaaages ago, but I realise it's probably better to ask in a public forum instead.
Is it technically possible to run sitespeed using Firefox, and as part of a test run, generate an output file for the scripted actions taken, that can be consumed by the firefox profiler tool listed below, maaaaaybe a bit a bit like how you can use Sitespeed to create Harfiles for later analysis??
If so, in high level terms how would you do it?
I'm asking in the context of wanting to automate test runs using the Firefox Profiler, but it's beyond my knowledge of both Sitespeed and of Firefox Profiler.
This talk about the power profiling tool for Firefox is quite a good intro https://fosdem.org/2023/schedule/event/energy_power_profiling_firefox/
As is the deck: https://fosdem.org/2023/schedule/event/energy_power_profiling_firefox/attachments/slides/5537/export/events/attachments/energy_power_profiling_firefox/slides/5537/FOSDEM_2023_Power_profiling_with_the_Firefox_Profiler.pdf