search

HTTPArchive / almanac.httparchive.org

HTTP Archive's annual "State of the Web" report made by the web community
https://almanac.httparchive.org
Apache License 2.0
611 stars 170 forks source link

Performance 2020 #905

Closed foxdavidj closed 3 years ago

foxdavidj commented 4 years ago

Part II Chapter 9: Performance

Content team

Authors Reviewers Analysts Draft Queries Results
@thefoxis @dimension85 @borisschapira @estelle @zeman @rviscomi @obto @noamr @ashrith-kulai @Zizzamia @exterkamp @max-ostapenko @dooman87 Doc *.sql Sheet

Content team lead: @thefoxis

Welcome chapter contributors! You'll be using this issue throughout the chapter lifecycle to coordinate on the content planning, analysis, and writing stages.

The content team is made up of the following contributors:

New contributors: If you're interested in joining the content team for this chapter, just leave a comment below and the content team lead will loop you in.

Note: To ensure that you get notifications when tagged, you must be "watching" this repository.

Milestones

0. Form the content team

1. Plan content

2. Gather data

3. Validate results

4. Draft content

5. Publication

dimension85 commented 4 years ago

I am interested in the reviewers role for this topic - Phil

borisschapira commented 4 years ago

I'm afraid I'm not fluent enough in English to create the content. I can help reviewing, though.

rviscomi commented 4 years ago

@thefoxis thank you for agreeing to be the lead author for the Performance chapter! As the lead, you'll be responsible for driving the content planning and writing phases in collaboration with your content team, which will consist of yourself as lead, any coauthors you choose as needed, peer reviewers, and data analysts.

The immediate next steps for this chapter are:

  1. Establish the rest of your content team. Several other people were interested or nominated (see below), so that's a great place to start. The larger the scope of the chapter, the more people you'll want to have on board.
  2. Start sketching out ideas in your draft doc.
  3. Catch up on last year's chapter and the project methodology to get a sense for what's possible.

There's a ton of info in the top comment, so check that out and feel free to ping myself or @obto with any questions!

To everyone else who has been nominated:

@logicalphase @Zizzamia @noamr @dimension85

we'd still love to have you contribute as a peer reviewer or coauthor as needed. Let us know if you're still interested!

rviscomi commented 4 years ago

@dimension85 @borisschapira thank you! I've added you both as reviewers.

estelle commented 4 years ago

I can also help out.

rviscomi commented 4 years ago

Thanks Estelle! I've also added you as a reviewer.

dimension85 commented 4 years ago

Excellent, thanks for adding me to the team - this is the first time I have been involved with this project and am looking forward to contributing.

logicalphase commented 4 years ago

Would love to co-author this section. Please let me know if I can help.

zeman commented 4 years ago

Happy to review again this year!

foxdavidj commented 4 years ago

An all-star team again this year. Really excited for this 🎉

noamr commented 4 years ago

On Wed, 1 Jul 2020 at 21:52 Rick Viscomi notifications@github.com wrote:

@thefoxis https://github.com/thefoxis thank you for agreeing to be the lead author for the Performance chapter! As the lead, you'll be responsible for driving the content planning and writing phases in collaboration with your content team, which will consist of yourself as lead, any coauthors you choose as needed, peer reviewers, and data analysts.

The immediate next steps for this chapter are:

  1. Establish the rest of your content team. Several other people were interested or nominated (see below), so that's a great place to start. The larger the scope of the chapter, the more people you'll want to have on board.
  2. Start sketching out ideas in your draft doc https://docs.google.com/document/d/1EeUJ88PS8Ms9XUrNIpM2tpm5Ad_K682ZMVfc24TM02w/edit?usp=sharing .
  3. Catch up on last year's chapter https://almanac.httparchive.org/en/2019/performance and the project methodology https://almanac.httparchive.org/en/2019/methodology to get a sense for what's possible.

There's a ton of info in the top comment, so check that out and feel free to ping myself or @obto https://github.com/obto with any questions!

To everyone else who has been nominated:

@hyperpress @Zizzamia https://github.com/Zizzamia @noamr https://github.com/noamr @dimension85 https://github.com/dimension85

we'd still love to have you contribute as a peer reviewer or coauthor as needed. Let us know if you're still interested!

Great! Would love to review.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/HTTPArchive/almanac.httparchive.org/issues/905#issuecomment-652589289, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAVT5QSPOAQUHH4N2SRL4TRZOAYLANCNFSM4OJ2CP2A .

ashrith-kulai commented 4 years ago

I would like to be a reviewer for this topic.

Zizzamia commented 4 years ago

Happy to join as a reviewer, looking forward to collaborate with the all Content team. 🌲 🚀 🌕

thefoxis commented 4 years ago

👋 hi everyone!

thanks again @rviscomi and @obto for selecting me to lead this effort. looks like we have a solid team of reviewers but we're short on analysts. should we be looking for people swapping areas of responsibility or look for new people to join?

I'm going to re-read last years chapter and reflect on what has happened in perf within the year. I think it'd be nice to speak to some new developments in the space, for example core web vitals or the shift in performance score algorithm. but it also depends on what sort of trends/data we're able to discover via the archive. I'll jot some notes down in the google doc within the next couple days. JavaScript is also always good to cover since it affects perf/UX greatly, so we could be looking at TBT & TTI. none of these were really covered last year so there would be no duplication (the previous report mentioned FID though, which is only relevant to people using RUM solutions/not Lighthouse).

if anyone has any ideas/suggestions, I'm all 👂

Zizzamia commented 4 years ago

@thefoxis yeah I agree 100% ! 🔥

Other points on top of my mind are:

What else? 🤔 One thing I am kind of curious to reflect is the different implications of TBT and LCP performance between SPA and SSR, and how websites are handling those performance trade-offs.

That's it for now, but I am sure the rest of the team has more angles we could deep dive 🌲🚀🌕

max-ostapenko commented 4 years ago

Count me in as an analyst ;)

foxdavidj commented 4 years ago

@max-ostapenko Added you as an analyst to the chapter :)

thefoxis commented 4 years ago

I added some notes to the doc clarifying my earlier points about lighthouse scoring and core web vitals. it's a bit hard to determine specific queries without seeing the data (I'll do more digging) but I think these will yield interesting findings.

I see a lot of questions surrounding the perf score, where it usually falls and now especially the new algorithm impact on the score. it's really a huge source of confusion to teams and I'd love to cover that first.

I have some reservations with core web vitals but I think with all the attention on them and their importance it'd be fascinating to uncover where those metrics usually land, especially by device type. I've been observing this data myself and you can see really fascinating findings in some cases.

@Zizzamia I also added your point about the element timing api as I think it ties nicely with general theme being new metrics and new way of measuring. do you have any further ideas and notes there? feel free to add to the doc :+1: we can also consider other subjects you've mentioned but I thought this particular one tied very obviously to the rest so I picked it first 😸

if anyone has any feedback or suggestions at this stage, let me know! 🙌

paulcalvano commented 4 years ago

@thefoxis - this looks great.

I believe most of the 2019 Performance chapter used Chrome User Experience Report data, but I agree that including Lighthouse scoring and web vitals would be great to include in the chapter.

A while back I wrote a discussion forum post on analyzing lighthouse scores at a high level, which you can see here. https://discuss.httparchive.org/t/analyzing-lighthouse-scores-across-the-web/1600

The lighthouse data in the archive contains all the details for each individual metric as well. Here's an example of @rviscomi extracting the scoring details for a single audit here - https://discuss.httparchive.org/t/how-and-where-is-document-write-used-on-the-web/1006.

Happy to help if you are struggling with what data is available. For Lighthouse in particular, sometimes it helps to browse a JSON report to get an idea of what can be queried

foxdavidj commented 4 years ago

@thefoxis You may have already, but here's a link to the metric results for last years performance chapter if you want to get an idea of what's possible: https://docs.google.com/spreadsheets/d/1zWzFSQ_ygb-gGr1H1BsJCfB7Z89zSIf7GX0UayVEte4/edit?usp=sharing

But also want to second what Paul said above. It's super helpful to look at the raw data that's being collected (and what isn't but should be). More than than happy to help you here as well.

foxdavidj commented 4 years ago

Hey @thefoxis, looks like things are moving along pretty smoothly. Is there anything you need from me to keep things moving forward, and have the chapter outline and metrics settled on by the end of the week?

dooman87 commented 4 years ago

Hi @obto and @thefoxis, I would like to get my hands dirty in the analysis if you are still looking :)

foxdavidj commented 4 years ago

@dooman87 marked you down as an analyst for the chapter :)

foxdavidj commented 4 years ago

@dooman87 We also have many other chapters still looking for analysts. If any of them interest you, we'd love to have you help out there as well!

thefoxis commented 4 years ago

@obto I think we'll be able to settle by end of the week. @rviscomi has been providing really helpful advice on what's possible 👍 I think we're close to finalising the outline.

thanks for the pointers, @paulcalvano 🙌

@Zizzamia can you add some information to the doc about one of your ideas, namely: Element Timing API and the potential of creating custom metrics? it's not 100% my area of expertise, so if you'd like to propose how to tackle getting some data here, that'd be great.

otherwise, we have plenty of data there already with core web vitals, LH scores and comparisons of FCP + TTFB to 2019 almanac 😇

jaisanth commented 4 years ago

I'm interested in being a reviewer, not sure if this is already filled up :-)

dooman87 commented 4 years ago

@thefoxis What's the plan to start with queries? Are we going to create a ticket for each metric in the doc? Sorry for noob questions - trying to get sense of the workflow :)

foxdavidj commented 4 years ago

@dooman87 The Chapter document is where each metric will be listed and progress will be tracked. Right now the next step is for analysts to look through all the metrics which have been proposed and verify they are possible to query.

Sorry for noob questions

No worries! In fact, @bazzadp just made a fantastic post about good next steps for analysts to take here: https://github.com/HTTPArchive/almanac.httparchive.org/issues/914#issuecomment-659205330 . I think you'd find it super helpful.

Also don't forget to join our #web-almanac slack so @paulcalvano can invite you to our Analysts channel. It's a great place to ask questions :)

Tiggerito commented 4 years ago

Hi, I'm an Analyst in the SEO chapter (so is your @max-ostapenko). I've been practising my SQL for a common area, Core Web Vitals by device. I got it mostly working and then checked your own and saw I'd gone the same way 👍

I thought it might be helpful in sharing? We're also wanting to look into this data by country. It should not need much rework to do that.

rviscomi commented 4 years ago

@dooman87 I've also written up a short summary of the recommended analysis workflow in the Analysts' Guide.

Hey @Tiggerito, yes it's a good idea to reuse queries whenever possible and this seems like a perfect opportunity for that. I'd suggest starting the draft PR described in the workflow above and adding the Core Web Vitals queries so we can take a look and add comments as needed.

Tiggerito commented 4 years ago

Hey @Tiggerito, yes it's a good idea to reuse queries whenever possible and this seems like a perfect opportunity for that. I'd suggest starting the draft PR described in the workflow above and adding the Core Web Vitals queries so we can take a look and add comments as needed.

Instructions made sense. Hopefully I got it right.

dooman87 commented 4 years ago

@dooman87 I've also written up a short summary of the recommended analysis workflow in the Analysts' Guide.

That's look awesome, thanks! Just to check my understanding, we are going to have one branch/PR where all analysts for particular chapter will work, right? It would make sense for me given we will split work and shouldn't step on each other toes.

Zizzamia commented 4 years ago

@thefoxis, I was reflecting on what we could monitor around Element Timing API and I think the best option we have is to take a couple of steps back and instead measure how many websites have the new PerformanceObserver in their JS minified files. Looking at this could help us understand how many websites are doing Performance Field Data measurements, which could be related to the use of some sort of field data library from Calibre, Web Vitals, Perfume.js, etc.

Then we can open up the conversation to other recent opportunities like measuring Element Timing API and we can mention all the other field data options.

@thefoxis @max-ostapenko @dooman87 do you think this is something we could add to the metrics?

thefoxis commented 4 years ago

@Zizzamia as I mentioned in the doc, I think at this point it's a bit late to add this to the scope since we're already working on analysing the data and I committed to a certain amount of work :) I'm happy to mention what was suggested in the doc but I don't want to commit to another sub-chapter at this point.

foxdavidj commented 4 years ago

@thefoxis @max-ostapenko @dooman87 for the two milestones overdue on July 27 could you check the boxes if:

Keeping the milestone checklist up to date helps us to see at a glance how all of the chapters are progressing. Thanks for helping us to stay on schedule!

Soham-S-Sarkar commented 4 years ago

I am interested in the reviewers role for this topic if we still have an open spot.

rviscomi commented 4 years ago

Hi @Soham-S-Sarkar. This chapter has 9 reviewers already, but I'll defer to @thefoxis if more are needed.

Meanwhile you can browse the chapter's outline to get a sense for the content direction.

rviscomi commented 4 years ago

@noamr @ashrith-kulai please request edit access to the planning doc to start your review of the initial outline.

thefoxis commented 4 years ago

thanks for interest @Soham-S-Sarkar but I think at this point we do not need more reviewers. it will be hard to process feedback with even more people involved.

@obto I won't be making any changes to the outline at this point, unless there are some metrics/data we can't obtain or the data proves non-conclusive/useful. the draft pr looks mostly complete but I defer to @max-ostapenko and @dooman87 to be able to provide more accurate progress status 👍

dooman87 commented 4 years ago

I still need to finish my requests related to LH performance score. Will work on them this week.

Tiggerito commented 4 years ago

Hi performers (is that right?),

The SEO chapter is wanting to get some Core Web Vitals data. You're web_vitals_by_device.sql covers the base of their requirements.

They are also interested in a report related a pages overall CWV score. I believe the overall score for a page is the worst case of good/ni/poor for all the factors.

Are you gathering CWV overall page scores at all?

foxdavidj commented 4 years ago

I've updated the chapter metadata at the top of this issue to link to the public spreadsheet that will be used for this chapter's query results. The sheet serves 3 purposes:

  1. Enable authors/reviewers to analyze the results for each metric without running the queries themselves
  2. Generate data visualizations to be embedded in the chapter
  3. Serve as a public audit trail of this chapter's data collection/analysis, linked from the chapter footer
thefoxis commented 4 years ago

@rviscomi @obto do we need to move some deadlines around because of the webpagetest snafu with throttling? since we'll be waiting for september data to come in, instead of finishing off with the august dataset. I can still maybe start writing some parts before the data is fully available but at the end of the day, we'll need to full analysis. thoughts?

tunetheweb commented 4 years ago

Why do we need to go forward to September? Why not go backwards to July?

Was Lighthouse 6 fully rolled out for July run? I think it was but could be wrong. Obviously you’d want that for this chapter so may have to wait if it was only half rolled out to the HTTP Archive crawlers for this run.

Are there any custom metrics for performance added for August run? If so they obviously won’t be there for July but can’t imagine there would be any for Lighthouse part anyway.

So maybe could get started on that and then rerun stuff for September to confirm your findings are still accurate and perhaps update the relevant stats for that?

tunetheweb commented 4 years ago

Though when did the bug come into play? I was assuming only for August crawl as that’s when it was noticed but maybe affected July too (is it a Lighthouse 6 bug?) in which case going back a month is obviously a non-runner.

dooman87 commented 4 years ago

Was Lighthouse 6 fully rolled out for July run? I think it was but could be wrong. Obviously you’d want that for this chapter so may have to wait if it was only half rolled out to the HTTP Archive crawlers for this run.

From what I can see in sample data, which I believe was generated from July run, there is a mix of version 5 and 6.

rviscomi commented 4 years ago

@bazzadp has a good point. @dooman87 is also correct that there's a mix of versions, but the majority of tests are using v6:

version count
6.1.1 4,506,136
6.1.0 1,825,717
5.6.0 16,600

The offending bug (https://github.com/WPO-Foundation/wptagent/pull/366) was merged on July 21 and AFAICT the July crawl ended on the ~23rd, so assuming the HTTP Archive agents were immediately updated (@pmeenan is that true?) the bug would have only affected a small proportion of the crawl.

@thefoxis I'll defer to you if you want to use the July crawl, which would contain some buggy and older-versioned Lighthouse data. If you opt for the September crawl, we can absolutely move the deadlines to a more realistic date.

pmeenan commented 4 years ago

Yes, the agents update the agent code hourly and the Lighthouse code daily.

thefoxis commented 4 years ago

@rviscomi I'm easy! I think I'd defer to you to make the decision on which dataset would be most reliable (since you have much more experience here than I do 😸 ). happy to go either way 👍

rviscomi commented 4 years ago

How about we pull only the v6 data from July, which would contain some of the unthrottled results (unless there's a way to detect and filter those out) and include that analysis in the initial draft, then we rerun the analysis when the September crawl is complete and update the draft as needed?