Reducing need for WPT API keys

[tkadlec]

Using grunt-perfbudget at the moment requires either a private instance of WebPageTest or an API key. The private instance is awesome, but WPT isn't exactly plug-and-play which is a barrier for many, particularly if the primary use is for budgeting.

The public key is great, but it does require emailing to request it, and then that key is only good for 200 page loads per day. Each run, both first and repeat views, count as a page load (10 runs, first and repeat view = 20 page loads). Depending on how you're budgeting and how many pages you're testing, that may or may not be enough. In addition, this slows down setup (as noted here: http://cognition.happycog.com/article/grunt-plugins-reviewed).

I was talking to @pmeenan about finding ways to simplify and speed up the process by removing the API key step. His suggestion was to perhaps allow the option for people to use AWS to accomplish the same thing. Basically, we could have a micro instance with WPT on it, as well as pre-configured agents. grunt-perfbudget would accept your Amazon API key, spin up the instances, and run the tests.

There'd be a minor cost associated through Amazon but you'd no longer have to get a WPT API key for each project, it would potentially speed up the time needed for setup, and it would also kill the daily load limit.

Good idea? Bad? Got a better one?

[thefivetoes]

Would it be possible to deploy to heroku instead of AWS? They recently created a one click deploy button for readme files, so you could easily config a private WPT instance that people could use? https://blog.heroku.com/archives/2014/8/7/heroku-button

[pmeenan]

I created an issue on the WPT side to track it and for discussions on how it might be implemented: https://github.com/WPO-Foundation/webpagetest/issues/310

As far as heroku goes, you'd still need EC2 for the test agents (Windows) so it makes sense to have the web UI over there as well. The UI should be able to run free-tier on a micro instance and the only costs would be associated with the time that the agent instances are running (a few cents per hour but they are only up while tests are running).

Once I have the WPT pieces in place grunt-perfbudget should be able to automatically create the UI instance and would just need the EC2 key and secret to be provided.

[lukefrake]

+1 I love this idea.

The small cost is negligible IMO.

[zeman]

I'm also looking at providing perfbudget support as a part of SpeedCurve.com for those that don't want the hassle of setting up and maintaining their own WPT agents and private instances.

You'd also get the data visualized overtime and each perfbudget test would add to the time series data giving you history and insight into how a project has evolved and when changes in the code base started to affect performance and blow out the budget.

Will be looking for some beta testers soon, let me know if you're keen.

[jaspervalero]

+1 on the AWS idea

[lukefrake]

Any more going on with this? I would love to get something like this up and running for my sites ASAP :)

If I can help, I will gladly!

[pmeenan]

Most of the plumbing is in place now and I should have an initial AMI for testing it out next week.

It will:

• Take an ec2 key and secret through user data
• Take a default API key through user data
• Optionally take an instance size to use for test agents
• Automatically update itself from trunk on github
• Automatically update the test agent code from the production WPT server
• Expose test locations for all of the EC2 regions
• Automatically spin up one test agent (on-demand) if a test is submitted and no agents are running for the location
• Every 5 minutes check to see if any EC2 agents need to be terminated:
  • If they are within 15 minutes of an hourly increment (since they are billed hourly, no reason to shut them down sooner)
  • and they haven't had work to do in the last 15 minutes
• If a bucket name is specified, tests will be archived to EC2 (possibly with a TTL). This will make the test results available even if you shut down the web server and spin a new one up later.

Still to do:

• Support more than one test agent for large batches of tests
• Support always keeping one instance running for a default test location
• Support spot instances or a hybrid of on-demand and spot
• Support deleting tests based on disk space (keep instance storage no more than 80% full)

[sergeychernyshev]

@pmeenan did you mean that tests will be archived to S3?

[pmeenan]

Oh yeah, mental typo, sorry On Oct 17, 2014 10:09 PM, "Sergey Chernyshev" notifications@github.com wrote:

@pmeenan https://github.com/pmeenan did you mean that tests will be archived to S3?

— Reply to this email directly or view it on GitHub https://github.com/tkadlec/grunt-perfbudget/issues/13#issuecomment-59595686 .

[pmeenan]

OK, an initial AMI is available that can auto-scale a single test agent for any of the regions where there are AMI's.

https://github.com/WPO-Foundation/webpagetest/blob/master/docs/EC2/Server%20AMI.md

I did some testing with the server on a micro instance and it was super-responsive so I think it will be a great low-cost and easy alternative for private instances. The lag for a new test agent coming online is a little annoying (5-10 minutes) but should be manageable.

The server auto-updates from trunk hourly and automatically pulls the latest agents from the public WPT so it should automatically pick up any fixes and features as we enhance it.

[tkadlec]

Awesome news @pmeenan!

[lukefrake]

Brilliant! Will be playing with this later on this week :+1:

[cfarm]

This is awesome!!

[kevindixon]

@pmeenan - gave it a shot, and able to spin-up a private instance really easily with a front-end, which run tests just fine. However, can't seem to get the API side to authenticate - getting locations works fine (no API key), but any requests requiring API key fail with '400 Invalid API Key'. The same requests work when directed to www.webpagetest.org (and a valid API key). User data:

ec2_key=<ec2_key>
ec2_secret=<ec2_secret>
api_key=myapikey
headless=0

Any idea how to debug? (Also, is there any documentation on the possible fields for user data - for instance, defining the S3 bucket?).

[Happy to move this over to WPO Foundation / webpagetest, if that is more appropriate]

[pmeenan]

I think I see the issue with configuring the API key and why it's not working. It is fixed now (just verified) but will only kick in for new instances (the settings are only run once when the instance is first started).

If you want to keep your existing instance but get the API keys working you can get the random API key that it generated (since it missed the configured one) by sshing into the instance and looking at /var/www/webpagetest/www/settings/keys.ini

It will be the last one listed and will have a description of "API key".

The fields that can be defined are all in the sample settings.ini file (for S3 you want the archives3* settings).

If you see any more bugs like with the API key or any enhancement ideas feel free to open issues over in the WebPagetest github project.

[kevindixon]

Thanks. That fixed it - one final question if I may.... what inbound ports need to open (other than 80)? I currently have them all open which I'd prefer not to...

[pmeenan]

Just 80 (and 22 if you want to be able to SSH in). The test agents talk to the server through the web interface so everything goes through there.