sitespeed.io 6.0 - Githubissues

soulgalore commented 7 years ago

In November I will be on leave from work for a week to make the foundation for sitespeed.io 6.0. The project has been running for 5+ years and it's time for the next major release. I got some ideas what will be included but I want your input too. I will only have a week so I can only do a couple of tasks :)

Here's my top list want I want to do:

Make it possible for plugins to add HTML #1141. The plugin structure we have today is really powerful but as a plugin writer you cannot add anything to the HTML output we should change that.
Release Coach 1.0 - we haven't given the coach any love for a while. We should update the advices and finally release 1.0.
Rewrite the documentation: We need more documentation where we talk about performance testing in general and it would be nice to change the structure and look and feel to make it easier to find things for users.
Use screenshot/filmstrip view and show them in the waterfall #1555 - It would be great to show screenshots when you hover the waterfall, we have the data, we probably need some help from Michael to do it though.
Show graphs to make some data easier to understand #870 - this is finished and just needs to get merged.
Rename tabs from the tools to what kind of data that you actually have in that tab. Change PageXray/Browsrtime etc to Waterfall/Video or something like that.
Move to NodeJS 8 (in October 8 will be LTS) and we can simplify some of the code. We could also make 6.0 using 8 and then rewrite the code later on.
Cleanup: Remove old Docker containers/Github repos to make it easier for people to find the right stuff.

Ok, that's what I have on top of my head. What do you think we should include in 6.0? @tobli @beenanner check if I missed something we've been talking about.

moehlone commented 7 years ago

If I got that right, sitespeed internally uses WPT API. So maybe it could be easily abstracted to drive iOS agents through WPT, like you said in Slack?

soulgalore commented 7 years ago

@moehlone yes you can do that right now, we push every config you add in --webpagetest.* to the WebPageTest NodeJS API so you can do whatever you want. However the NodeJS wrapper is lagging behind the WPT API but I think it's nothing special that needs to be done for WPT.

moehlone commented 7 years ago

Okay, so maybe this could be a good step by step guide for the documentation. Thanks!

soulgalore commented 7 years ago

Yeah can you give feedback what's missing: https://www.sitespeed.io/documentation/sitespeed.io/webpagetest/ (sometimes it's hard to see things when you write them yourself :))

siteriaitaliana commented 7 years ago

What about a plugin to pull in new relic nrql results? Also wpt native support might be good the whole setup of pointing an external instance is pretty flaky right now, keep getting runs errors and test being queued due to slow to refresh php queues or when the fun result calculation in Wpt gets corrupted. Last yes the Lighthouse support is something useful as the brosertime and coach timing metrics are super fast due to headless browser runs. Thanks to collect ting feedbacks and I promise to find some time to help in, L.@siteriaitaliana

soulgalore commented 7 years ago

thanks @siteriaitaliana !

I think the overall strategy is to make everything pluggable but not add more integration in sitespeed.io. So if you wanna pull in data from New Relic or Lighthouse that will work, but you need to find the plugin or create it yourself (that is possible in 5.x but we miss the possibility to add HTML, but that will come in 6.0). We gonna move out GPIS to a standalone plugin and only keep WebPageTest for 6.0. Making it easier to integrate your own plugin is the goal for 6.0 :)

For WebPageTest I'm not sure exactly what you mean. Do you run your test on WebPageTest.org or have your own instance? We've been using our own private instance at work using AWS and that works ok for us. WebPageTest.org is flakey but I mean you shouldn't use that if you care about metrics, then you need to have your own instance. Or run Browsertime with --video & --speedIndex if Chrome/Firefox is enough for you.

siteriaitaliana commented 7 years ago

Thanks for your prompt answer @soulgalore

Would be interesting to find a way to have a plugin directory somewhere to avoid re-building a plugin that may be somebody has already built (RE.. new relic).

About WPT, we are using a private own instance in AWS but at some point I start to lose runs due to timeout (test get stuck in the pearstack queue in some ways and only recover when the queue gets cleaned up) or as I'm trying to debug right now with a "Sorry, that test location already has too many tests pending. Pleasy try again later." error possibly related to this https://www.webpagetest.org/forums/showthread.php?tid=11007

Problem with browsertime is that the timing metrics are terribly slow, as I've mentioned before, not sure why, probably because is using headless browser.

Did you experience any of these?

UltraNemesis commented 7 years ago

Resolving #1686 would be great. I think this is a very important feature.
Coach output from the WPT har would be nice.
Flimstrip and video in WPT result.

soulgalore commented 7 years ago

@siteriaitaliana for WebPageTest: no haven't got that error (we've been running for two years now. we have other errors/problems but not that one :)). We have our setup described here https://wikitech.wikimedia.org/wiki/WebPageTest

Problem with browsertime is that the timing metrics are terribly slow

No? Slow in the metrics are slow or slow to collect? Can you create a separate issue for that and we can try to help you there. We use c4.large on AWS (both for WebPageTest and Browsertime/sitespeed.io). At least for me that seems the (minimum) size you should use. We use DO on dashboard.sitespeed.io but we don't get the stable metrics as we get on AWS.

soulgalore commented 7 years ago

@UltraNemesis thanks for the feedback

Resolving #1686 would be great. I think this is a very important feature.

I did a POC for that today and need to sync with @tobli and @beenanner

Coach output from the WPT har would be nice.

Yes that would be cool. I need to try out if we can just push it as custom JavaScript to WebPageTest.

Filmstrip and video in WPT result.

I started with the video but the current NodeJS API gave me some problem but maybe there's a fix for that.

xavier-medioni commented 7 years ago

Resolving #1634 about hardcoded xvfb configuration, and #1418 about cumulative budget would help using sitespeed out of docker context (yes, shame, I know...), as node library, for instance :-)
Maybe adding options to be able to retrieve data from Google Chrome profiling tools (https://github.com/paulirish/automated-chrome-profiling -> https://github.com/paulirish/automated-chrome-profiling/blob/master/get-timeline-trace.js) Thanks by advance !

soulgalore commented 7 years ago

Hey @xavier-medioni thanks for your input!

hardcoded xvfb configuration,

Can you show exactly how you need it be done and I can have a go.

cumulative budget would

That I would solve right now by creating a new sitespeed object between runs?

Maybe adding options to be able to retrieve data from Google Chrome profiling tools

You can get the trace log from Chrome today --browsertime.chrome.traceCategories but we don't do anything with the data (except creating the HAR). Using of Paul Irish libraries to parse would be fine, I think that be a good plugin when 6.0 lands, so you as plugin creator can create HTML output.

xavier-medioni commented 7 years ago

Hi performance buddies !

A few days ago, I found a very useful feature on chrome-har-capturer tool. It extends the HAR file with initiator data, such as:

"_initiator": {
                    "type": "script",
                    "stack": {
                        "callFrames": [
                            {
                                "functionName": "send",
                                "scriptId": "22",
                                "url": "https://url.to.script.file.js",
                                "lineNumber": 127,
                                "columnNumber": 5
                            }
                        ]
                    }
                }

or, according origin:

"_initiator": {
                    "type": "parser",
                    "url": "https://url.to.page.com/",
                    "lineNumber": 900
                }

It is awesome when we "track the trackers", and we try to associate a request to a script (even if it does not cover every usecases) and could be a good base for interesting plugins.

Is it "too much Chrome relative" :-) ?

Thanks by advance !

Xavier

tobli commented 7 years ago

Hi! We have a PR for that here: https://github.com/sitespeedio/chrome-har/pull/6. There are a few tweaks needed from the review, but it hasn't seen any activity. It'd be awesome if you'd like to adopt the PR and tweak it!

Thanks, /Tobias

soulgalore commented 7 years ago

6.0 is released, let us do the same for 7.0.

sitespeedio / sitespeed.io

sitespeed.io 6.0 #1719