sitespeedio / sitespeed.io

sitespeed.io is an open-source tool for comprehensive web performance analysis, enabling you to test, monitor, and optimize your website’s speed using real browsers in various environments.
https://www.sitespeed.io/
MIT License
4.73k stars 601 forks source link

How to get CPU usage from Firefox when you run sitespeed.io tests? #3944

Open mrchrisadams opened 1 year ago

mrchrisadams commented 1 year ago

Your question

Hi folks.

I think I asked this in slack aaaaages ago, but I realise it's probably better to ask in a public forum instead.

Is it technically possible to run sitespeed using Firefox, and as part of a test run, generate an output file for the scripted actions taken, that can be consumed by the firefox profiler tool listed below, maaaaaybe a bit a bit like how you can use Sitespeed to create Harfiles for later analysis??

If so, in high level terms how would you do it?

I'm asking in the context of wanting to automate test runs using the Firefox Profiler, but it's beyond my knowledge of both Sitespeed and of Firefox Profiler.

This talk about the power profiling tool for Firefox is quite a good intro https://fosdem.org/2023/schedule/event/energy_power_profiling_firefox/

As is the deck: https://fosdem.org/2023/schedule/event/energy_power_profiling_firefox/attachments/slides/5537/export/events/attachments/energy_power_profiling_firefox/slides/5537/FOSDEM_2023_Power_profiling_with_the_Firefox_Profiler.pdf

mrchrisadams commented 1 year ago

I vaguely remember some of this now, and profiler docs are helpful too.

The output that Firefox Profiler needs is generated by the Gecko Profiler, a piece of c++ code inside Firefox.

If you have the ability to run javascript in the context of driving the browser, there's a profiler object you can interact with.

Here's the code from the corresponding docs profile page:

(() => {

  // I think these are the settings telling the profile what to profile, how often to sample, and 
  // setting a limit of how many samples to create.
  const settings = {
    entries: 1000000,
    interval: 0.4,
    features: ["js", "stackwalk", "threads", "leaf"],
    threads: ["GeckoMain", "Compositor"]
  };

  // start the profiler. and recording stuff
  Services.profiler.StartProfiler(
    settings.entries,
    settings.interval,
    settings.features,
    settings.features.length,
    settings.threads,
    settings.threads.length
  );

  // dump the output somewhere, currently to the console I guess?
  setTimeout(() => {
    Services.profiler.getProfileDataAsync().then(profile => {
      for (let i = 0; i < profile.threads.length; i++) {
        const thread = profile.threads[i];
      }
      console.log(profile);
      // Stopping the profiler will delete the data in the buffer.
      Services.profiler.StopProfiler();
    });
  }, 500);
})();

I can't remember if sitespeed lets you run javascript in this context easily, or how best to write this to a file, but presumably, if you know how to do these two things, then it's not a significant leap to be able to consume them in the Profiler, or even expose some metrics for presentation inside sitespeed.

mrchrisadams commented 1 year ago

I think Browsertime might be able to do this, which Sitespeedio uses. If you can use Selenium to access the browser console as part of a test run, then I think you might be able to execute the code above using Browsertime, based on what Browsertime already lets you do:

Browsertime uses Selenium NodeJS to drive the browser. It starts the browser, load a URL, executes configurable Javascripts to collect metrics, collect a HAR file.

To get the HAR from Firefox we use the HAR Export Trigger and Chrome we use Chrome-HAR to parse the timeline log and generate the HAR file.

More below:

https://www.sitespeed.io/documentation/browsertime/introduction/

I don't know what overhead any of this would add on top of profiling. Still, it makes me think this is doable.

soulgalore commented 1 year ago

Hi @mrchrisadams we support getting the profile using --browsertime.firefox.geckoProfiler. If you want to run it exactly as in the example it would look like:

sitespeed.io -b firefox --browsertime.firefox.geckoProfiler --browsertime.firefox.geckoProfilerParams.features "js, stackwalk, threads, leaf" --browsertime.firefox.geckoProfilerParams.thread  "GeckoMain, Compositor" https://www.sitespeed.io -n 1

That will generate a geckoProfile-1.json.gz file, the problem is I don't fully understand the internal Firefox format, how do I see the energy consumption? I could build something parses the file but I need some documentation/hints how to get the juicy bits out of the file.

mrchrisadams commented 1 year ago

Oh sweet, I had no idea this was already supported Peter! I'm less familiar with the internal Firefox Format, but this issue here seems to be the one we'd want to watch, or ask questions about in meantime:

https://github.com/firefox-devtools/profiler/issues/4673

mrchrisadams commented 1 year ago

I've asked Florian at Mozilla about this - he's the author of that issue and has been the person I've been corresponding with.

mgifford commented 1 year ago

Ok, so what this would allow you to do is set the specifics of the Firefox browser that you are using to test with sitespeed.io by specifically configuring it with something like --browsertime.firefox.geckoProfiler averageUser.json or something like that. 

Then when you're crawling a site, you can use the specifications of that Firefox browser, rather than whatever the default is when running Firefox as the rendering engine. 

So something like this would work if you wanted to use that Firefox profile to evaluate 50 web pages with the SWD sustainability model:

docker run --rm -v "$(pwd):/sitespeed.io" sitespeedio/sitespeed.io:29.3.0 https://example.com -d 3 -m 50 -n 2 --sustainable.enable --axe.enable --details --description --browsertime.videoParams.filmstripQuality 5 --sustainable.model swd --firefox --browsertime.firefox.geckoProfiler averageUser.json
canova commented 9 months ago

Here's a documentation to the profile file format: https://github.com/firefox-devtools/profiler/blob/main/src/types/profile.js and some some more documentation here: https://github.com/firefox-devtools/profiler/tree/main/docs-developer

canova commented 9 months ago

Hey folks, we had an in-person chat with @mrchrisadams 2 days ago and I was made aware of the issue where browsertime fails when the "power" feature is set to capture power measurements.

After spending some time on it, I managed to reproduce and find the cause of issue. It looks like it's because of the geckodriver build, we both had the arm64 macbooks, and looks like the sitespeedio/geckodriver library uses the x86_64 binaries all the time.

After https://github.com/sitespeedio/geckodriver/pull/14 this issue should be fixed. With this PR applied, I can successfully power profile with the sitespeed.io tool.

Here's the complete command I used, it's the exact replication of our "power" preset inside Firefox:

sitespeed.io -b firefox --browsertime.firefox.geckoProfiler --browsertime.firefox.geckoProfilerParams.bufferSize 134217728 --browsertime.firefox.geckoProfilerParams.interval 10 --browsertime.firefox.geckoProfilerParams.features "screenshots,js,stackwalk,cpu,processcpu,nostacksampling,ipcmessages,markersallthreads,power" --browsertime.firefox.geckoProfilerParams.threads "GeckoMain,Renderer" https://www.sitespeed.io -n 1

Here's an example profile output I got with this command: https://share.firefox.dev/3MWPGXZ

Note that this command will still fail on arm64 macOS machines until we land the sitespeedio/geckodriver PR and publish a new version (and start using that version in both browsertime and sitespeed.io tools).

On the otherhand, I agree that we could make this easier by having these sensible defaults for power profiling in a json file like power.json.

mrchrisadams commented 9 months ago

Oh sweet, thanks @canova!

Let me check if I understand.

We might define the params as a config file, power.json, like so:

{
 "browsertime": {
    "browser": "firefox",
    "iterations": 1,
    "firefox": {
      "geckoProfilerParams": {
        "bufferSize": 134217728,
        "interval": 10,
        "features": "screenshots,js,stackwalk,cpu,processcpu,nostacksampling,ipcmessages,markersallthreads,power",
        "threads": "GeckoMain,Renderer"
      },
    }
  }
}

Which would mean we could then run sitespeed like this:

sitespeed.io https://example.org --config power.json

I think with the PRs mentioned above merged in, we'd have a way profile visiting a page, in an automated fashion check for regressions, and ideally get direct power measurements, rather than relying on modelled figures in SWD for end user devices.

Profiling a more complicated user journey.

Presumably, once we have that, it's then possible to run more compicated user journeys too. They're documented in more detail on the page below:

https://www.sitespeed.io/documentation/sitespeed.io/scripting/#measure-one-page-after-you-logged-in

For the purposes of this convo tho, I think you could have a file called login.js containing the code below:


export default async function (context, commands) {
  await commands.navigate(
    'https://example.org'
  );
  try {
    // Find the sign in button and click it
    await commands.click.byId('sign_in_button');
    // Wait some time for the page to open a new login frame
    await commands.wait.byTime(2000);
    // Switch to the login frame
    await commands.switch.toFrame('loginFrame');
    // Find the username fields by xpath (just as an example)
    await commands.addText.byXpath(
      'peter@example.org',
      '//*[@id="userName"]'
    );
    // Click on the next button
    await commands.click.byId('verifyUserButton');
    // Wait for the GUI to display the password field so we can select it
    await commands.wait.byTime(2000);
    // Wait for the actual password field
    await commands.wait.byId('password', 5000);
    // Fill in the password
    await commands.addText.byId('dejh8Ghgs6ga(1217)', 'password');
    // Click the submit button
    await commands.click.byId('btnSubmit');
    // In your implementation it is probably better to wait for an id
    await commands.wait.byTime(5000);
    // Measure the next page as a logged in user
    return  commands.measure.start(
      'https://example.org/logged/in/page'
  );
  } catch(e) {
    // We try/catch so we will catch if the the input fields can't be found
    // We could have an alternative flow ...
    // else we can just let it cascade since it caught later on and reported in
    // the HTML
    throw e;
  }
};

And then run the commmand, including those steps in an automated fashion, as well as the power.json config:

sitespeed.io https://example.org --config power.json --preScript login.js 

Is this how you'd see it too?

Where I need help understanding all this so we could close the issue

The main thing I think I am missing is how to convert the --browsertime.firefox.geckoProfiler flag to the equivalent for placing into power.json. That's something I think @soulgalore might have some insight on.

Once we have that, I think I'd understand this issue well enough to have a go at contributing it to the sitespeed docs, once the necessary PRs are merged in.

@mgifford this might interest you too :)

soulgalore commented 9 months ago

@canova so what I meant (or what I need), is that I need to understand the correct way to calculate the CPU usage? Summarising all threadCPUDelta for a profile? And then the unit differs per platform? Thinking the easiest way for getting usage, would be that we iterate over the geckoprofile output and generate one new metric, I can fix that as long someone help me with the correct way to calculate it :)

canova commented 9 months ago

@mrchrisadams Oh, I wasn't aware of the --config option already. Yes, this config json file makes things a lot easier with the way you use it.

For --browsertime.firefox.geckoProfiler, I think you should be able to do something like:

{
 "browsertime": {
    "firefox": {
      "geckoProfiler": true,
      ...
    }
    ...
  }
}

(with the other options omitted)

@soulgalore

Ah I see. This threadCPUDelta values show you the CPU usage and not the power usage values. They mostly correlate to each other but not exactly the same. But here's a good thing, you don't really need to extract CPU usage value because browsertime already has CPU usage information in ms unit here: https://github.com/sitespeedio/browsertime/blob/4547991bf0ebed6692f01ad7c94997652804aa1d/lib/firefox/webdriver/firefox.js#L179-L189 It doesn't need to have the profiler active, so it will be present all the time. I don't know if you display this already. We use this metric to track cpu usage in Firefox perf testing (the variable is called powerusage here, which is a bit misleading though).

If you want to extract the power usage values, I think it's possible to extract a single energy usage value out of the profile data (but not as easy as the CPU usage). Let me try to explain how to extract this value of a profile json.

  1. Unzip the json file if it's zipped already and load it into memory
  2. Look at the profile.meta.configuration.features array and see if it contains "power" feature in the array. If it doesn't contain this feature it means that the power values are not recorded, so you can bail early.
  3. If there is a "power" feature, great! Then we can continue with extracting this information.
  4. All the power values are stored as "counters" in the profile data. The counters are stored directly at the top level of the profiles, so you can access it like profile.counters. But there are also other counters for other purposes. So iterate over these counters and find the ones with category: "power".
  5. Now we have all the power counters, we can iterate over them and accumulate to have a single value. We already have a similar logic in the profiler codebase here: https://github.com/firefox-devtools/profiler/blob/7e2bb7412958fe80c1982384a1a1502115a7eb73/src/components/tooltip/TrackPower.js#L50-L69 This function takes a start and end time and calculated the value for a single counter. What you need to do is to calculate this with every values in the counter (so no start/end time needed), and repeat that process for every other power counters to accumulate all.
  6. There is also a function to format this power value here: https://github.com/firefox-devtools/profiler/blob/7e2bb7412958fe80c1982384a1a1502115a7eb73/src/components/tooltip/TrackPower.js#L95-L137 You can convert that value to co2 consumption like this.

Hope this helps! Also thanks for publishing all the packages with https://github.com/sitespeedio/geckodriver/pull/14 . @mrchrisadams you should be able to capture power profiles with the sitespeed.io and browsertime tools now with the latest version.

soulgalore commented 8 months ago

Thanks @canova that is super helpful! I started this morning to implement it, I'm almost done but I don't fully understand the counters data structure. What do the two numbers in each data array represent? Which one should I summarise?

{
  name: 'Power: DRAM',
  category: 'power',
  description: 'RAPL DRAM',
  sample_groups: [ { id: 0, samples: { schema: { time: 0, count: 1 },
  data: [
    [ 3759.491881, 777833777533 ],
    [ 3770.838823, 3017850 ],
    [ 3779.618929, 542534 ],
    [ 3790.957486, 2577040 ],
    [ 3800.470425, 2475315 ],
     ... } ]
}
canova commented 8 months ago

@soulgalore Nice! We use schema to identify which data field corresponds to which value. If you look at the line above, you'll see this:

schema: { time: 0, count: 1 }

It means that first number is time and the second number is count. And count is the one you're looking for. But instead of hardcoding 1 directly, I would recommend getting that from schema, like:

const countIndex = samples.schema.count;
for (const datum of samples.data) {
    const count = datum[countIndex];
    <...accumulate...>
}

This way it will be more future-proof in case of any change we make in the json format.

Also, it's good to mention that this counter object is changing slightly in Firefox 122 (see bug 1866629, currently in beta, will ship to release on January 23) We are removing sample_groups array and keeping only one samples object instead. Previously that sample_groups were implemente thinking that we might need it in the future and it only contained a single samples object. This bug I mentioned removes it since we never really needed it so far. See the new object type in here too. To be able to handle these differences, we use version numbers in our json format. You can find that number in profile.meta.version. We also have a changelog here. So if it's version >= 29, it won't have sample_groups and and if it's version < 29 it will have it. Sorry for this annoyance, we usually keep the json format stable but make some changes rarely.

soulgalore commented 8 months ago

Thank you @canova ! Ok, that works fine. What do we call this metric? We have "powerUsage" today from Android phones (that's info that we get from dumpsys batterystats). I need fix so the metric is reported from browsertime. I'm thinking reporting the raw usage and then in sitespeed.io we can use co2 to convert it.

canova commented 8 months ago

Hm, I'm not so sure about the name. We simply call them "power" in the Firefox Profiler but already having "powerUsage" makes it trickier.

soulgalore commented 8 months ago

I pushed a PR in Browsertime calling it powerConsumption, let me know if anyone wants to change the implementation or naming :)

When that rolls out, I can add it to sitespeed.io so we use CO2 to convert the Wh.

soulgalore commented 8 months ago

Ok, it will look like this:

browsertime --config power.json https://www.sitespeed.io -n 1

I added a flag that needs to be set including the "power" in features --firefox.powerConsumption.

{
  "browser": "firefox",
  "iterations": 1,
  "firefox": {
    "powerConsumption": true,
    "geckoProfiler": true,
    "geckoProfilerParams": {
      "features": "power"
    }
  }
}