Build automatic sandbox testing on pull request

[quicksketch]

Since Backdrop won't work easily with Simplytest.me, and since we're on a platform that enables this more easily than before, we should look into building an automated sandbox environment for each pull request, similar to Lullabot's setup: http://www.lullabot.com/blog/article/github-pull-request-builder-drupal

However that might be a bit too liberal, we should expect hundreds of pull requests to be open at all times, which could cause some scalability issues. If possible, maybe we could make it optional if a sandbox were needed somehow, like assigning a label on the associated issue. Note that since pull requests aren't in the "issues" repository, we can't assign labels to them.

[sirkitree]

Well, I always say that if your project gets to the point of having a scalability problem, you at least have some sort of indication of success. :wink: But I'm a little unclear since it sounds like you're talking about core (backdrop) and not the development of the companion website (backdropcms.org).

I think this could easily be setup on a $20/mo Linode machine to start with and since I have ulterior motives for having more eyes on and use cases for tugboat (our current pet name for the collection of jobs described in that article) I would be happy to get this setup (or help enlist someone else to help). It can certainly be customized to work on any number of criteria since it's really just automated bash scripts.

I'm not too terribly worried about resources since the biggest drag on most of our projects is copying canonical databases, feature reverts and file copying, mostly none of which we'd need to worry about for core. I think there would be plenty of room for crafting this in an efficient manner to handle quite a lot of pull requests against not only the core project but cms site as well.

[quicksketch]

But I'm a little unclear since it sounds like you're talking about core (backdrop) and not the development of the companion website (backdropcms.org).

I put this in the backdropcms.org queue because I imagine this will require setting up separate code and servers (rather than adding code to Backdrop itself). This queue probably isn't really the right place either. shrugs

I would be happy to get this setup (or help enlist someone else to help).

I've invited you to a new Linode account that has a box waiting to be spun up. Personally, I prefer CentOS but I can swing Ubuntu also if that's what you're inclined to use.

I think there would be plenty of room for crafting this in an efficient manner to handle quite a lot of pull requests against not only the core project but cms site as well.

Great!

[sirkitree]

Well. I appreciate the trust here, but it's been a long time since I've setup and configured a new server. I'm not sure I trust myself to do a very good job with that portion. :frowning:

If I had one ready to go with Jenkins installed I'd be much more comfortable getting a pull request builder underway.

I'll see if I can get some help from Ben tomorrow. Perhaps it's gotten even easier on Linode and I'm worried about nothing.

[quicksketch]

No problem. I can set it up and then turn it over to you. :)

[sirkitree]

Do we need this since https://dashboard.getpantheon.com/products/backdrop/spinup - or am I misunderstanding how that's going to be used?

[quicksketch]

Do we need this since https://dashboard.getpantheon.com/products/backdrop/spinup - or am I misunderstanding how that's going to be used?

The Pantheon spin-up is more for demos, and will only be used on the latest stable release of Backdrop, rather than be a per-issue testing environment. I think Pantheon is primarily interested in getting users to try out the entire service and sign up for accounts, rather then hosting temporary sandboxes.

We've had @Gormartsen volunteer to take on setting this up on a new server his company will be providing. We've set up a new repo at https://github.com/backdrop/qa.backdropcms.org to hold the code.

I had been thinking that we would use Lullabot's https://github.com/Lullabot/jenkins_github_drupal repository to build out this infrastructure, but since that project has largely been abandoned in favor of http://tubboat.qa, which is not open-source, we're back in a situation where there isn't a supported tool for doing this sort of thing.

I think Gor's plan is to use the Lullabot scripts for inspiration and reference, but build out something loosely based on the concepts. I'll let him post his full plan here when he gets the chance.

[sirkitree]

Tugboat's goal is to be open source once we get our :hankey: together. There will be a UI component that will be the part we try to monetize.

That being said, there is some interesting work going on using Docker with it, and I've sent the team an email asking if they think this is something Tugboat could even handle, so we'll see what they think.

I personally think it would be nice to have something like this for Backdrop, but also possibly use it as a means of cross promotion for both projects.

[quicksketch]

Yeah, I'd love to see tugboat come to fruition. Last I checked with James at Lullabot, it seemed like it wasn't ready to public display (and that was 18 months ago). Since we manage all Backdrop's code repositories publicly, it didn't look like it would be an option in the near future.

[Gormartsen]

Sorry I lost track on this issue. Right now I almost finished git.lc project. It is capable to setup new website like pr-NNNN.qa.backdropcms.org based on PR.

Will you be ok to use it to handle this issue?

I will configure it then.

PS. Is 'Gormartsen was unassigned by quicksketch on Aug 14, 2014' correct date? Sounds like some timestamp issue for github.

[Gormartsen]

@quicksketch @jenlampton This task is ready to go production.

Example how it works now. In my fork: https://github.com/Gormartsen/backdrop

You can see file: https://github.com/Gormartsen/backdrop/blob/1.x/.gitlc.yml

deploy:
  branch:
    server: backdrop.vps-private.net
    username: qa
    dir: '{home}/github/branch/{branch}'
    env_vars:
      domain: '{branch}.qa.backdrop.vps-private.net'
      site_name: '{branch} testing site'
      site_mail: 'testing@{branch}.qa.backdrop.vps-private.net'
      account_name: admin
      account_mail: 'testing@{branch}.qa.backdrop.vps-private.net' 
    scripts:
      init: '{deploy_dir}/core/scripts/init_deploy.sh'
    webhooks:
      init: http://github.backdrop.expert/webhook/init
      remove: http://github.backdrop.expert/webhook/remove
  pull_request:
    server: backdrop.vps-private.net
    username: qa
    dir: '{home}/github/pr/{pr_number}'
    env_vars:
      domain: '{pr_number}.qa.backdrop.vps-private.net'
      site_name: '{pr_number} testing site'
      site_mail: 'testing@{pr_number}.qa.backdrop.vps-private.net'
      account_name: admin
      account_mail: 'testing@{pr_number}.qa.backdrop.vps-private.net' 
    scripts:
      init: '{deploy_dir}/core/scripts/init_deploy.sh'
    webhooks:
      init: http://github.backdrop.expert/webhook/init
      remove: http://github.backdrop.expert/webhook/remove

It's http://git.lc integration config file.

It deploys a website for each branch and pr.

branch part

I created branch test. Gitlc created live site http://test.qa.backdrop.vps-private.net Deploy log is here: http://git.lc/Gormartsen/backdrop/deploy/deploy-Gormartsen_backdrop_test-4715 You can find it by clicking 'details' for deploy status.

pull request part

I created PR 10. https://github.com/Gormartsen/backdrop/pull/10

Gitlc created live site http://10.qa.backdrop.vps-private.net/ Deploy log is here: http://git.lc/Gormartsen/backdrop/deploy/deploy-Gormartsen_backdrop_10-4717

You can find it by clicking 'details' for deploy status.

Or see it directly here: http://git.lc/Gormartsen/backdrop

Removing

When branch or PR deleted, website get deleted from server as well. Example: https://github.com/Gormartsen/backdrop/pull/9 Status provided: gitlc/undeploy/pr-9 — I get job satisfaction.

Other details

You can see webhook pointed to http://github.backdrop.expert http://github.backdrop.expert via webhook and qa modules create and destroy domains, database name and user on server backdrop.vps-private.net. See code here: https://github.com/backdrop-ops/github-auto/tree/master/modules/webhook https://github.com/backdrop-ops/github-auto/tree/master/modules/qa

You can find login and password in deploy log, but we can change so in init_deploy script: https://github.com/Gormartsen/backdrop/blob/1.x/core/scripts/init_deploy.sh

#!/bin/sh
# This script create a website based on ENV variables provided by qithab-auto

#set site path
SITEPATH="$HOME/domains/$DOMAIN"

# Go to domain dir
cd $SITEPATH

#link backdrop files
ln -s $GITLC_DEPLOY_DIR/* ./
ln -s $GITLC_DEPLOY_DIR/.htaccess ./

#install backdrop
php $GITLC_DEPLOY_DIR/core/scripts/install.sh --root=$SITEPATH --db-url=mysql://$DATABASE_USER:$DATABASE_PASSWORD@localhost/$DATABASE_NAME --account-mail=$ACCOUNT_MAIL --account-name=$ACCOUNT_NAME --account-pass="$DATABASE_PASSWORD" --site-mail=$SITE_MAIL --site-name="$SITE_NAME"
echo "USER: $ACCOUNT_NAME PASS: $DATABASE_PASSWORD"

This script can be a part of repository or can be uploaded on backdrop.vps-private.net server separately. If it uploaded separately, proper path need to be set into .gitlc.yml
deploy->branch-> scripts->init deploy->pull_request-> scripts->init

Let me know if it is exactly what you need and I give you instructions how to hook up backdrop/backdrop to http://git.lc and http://github.backdrop.expert

backdrop.vps-private.net - is VPS created to provide websites for this task. Drop me a public key and I give you SSH access to qa@backdrop.vps-private.net

Also register on http://dh.it-patrol.com and give me loginname. I will link qa@backdrop.vps-private.net to this account and you will be able to work with hosting panel.

Example of hosting panel resource usage page:

[quicksketch]

Whoa! There are a lot of great things in here!

I've got a lot of questions!

• git.lc looks like a very interesting tool that could be handy! We've talked about this exact feature, though I can't find the issue for it... very impressive looking!
• It looks like git.lc is about code review assistance, rather than live testing? Or is it planning to offer both?
• Is this code available as Open Source somewhere? Although we're using a lot of non-open tools (GitHub itself for example, Travis-CI), I'd love to see Backdrop encourage open tooling.
• On the "details" link itself, the log is unnecessarily (IMO) filled with the information about downloading the repository. 9/10 times I could see myself clicking this link just to get the admin user/pass. Could we make that info available at the top of the file (or even in the description instead of "I get job satisfaction.")
• How do we get to the test site from the "details" link? I couldn't find it when looking through the log. How could we make it more clear?
• It looks like this could be used on both core and contrib, correct?
• Are you sure you want to unconditionally spin up a sandbox on every PR? What happens when PRs are updated via a git push -f?

Will you be ok to use it to handle this issue?

If you provide the information for hooking up the webhooks, I can finish the job if needed. Though I'd like to get some more feedback from others before we push this out. It's very exciting work! Nice job @Gormartsen!

[Gormartsen]

Tnx!

Answers:

It looks like git.lc is about code review assistance, rather than live testing? Or is it planning to offer both?

Git lc is my business project. I was starting it to automate code style checks. Only later I figure out that I can do deploy via it. So yes, both. I am looking to add testing maybe as well.

Is this code available as Open Source somewhere?

No, it is not. How I said, it's business project. I am planning to launch it public free to use for open source this summer (like Travis) but with small price for private repo/business companies. Also I want to switch it to backdrop from d7.

On the "details" link itself, the log is unnecessarily (IMO) filled with the information about downloading the repository. 9/10 times I could see myself clicking this link just to get the admin user/pass.

I will update look and feel for deploy log page. But we can also add a message 'test site created. Link, login,pass' to PR and related issues via githubauto webhook part.

Are you sure you want to unconditionally spin up a sandbox on every PR?

I setup VDS to spin up all prs. It's 18gb disk space with 2.4 cpu and 4gb ram. Properly configured to be fast an handy. It should be enough for a while.

What happens when PRs are updated via a git push -f?

Code will be updated.

When we get drush ready or even our own replacement, we can add scripts->after to call 'drush -y updb' every time after deploy changes.

[Gormartsen]

If you provide the information for hooking up the webhooks, I can finish the job if needed.

Check this issue:

https://github.com/backdrop-ops/backdropcms.org/issues/181

I created a website github.backdrop.expert to run code from backdrop-ops/github-auto repo.

I think we need to change it to qa.backdropcms.org or something like this.

Webhook part you will find in hook_webhook part of qa module in this repo.

Also any module that implements hook_webhook can be a part of deploy processing.

So check it as well and ask me questions if needed.

[Gormartsen]

It looks like this could be used on both core and contrib, correct?

It is. I will write an example to use it with contrib.

[Gormartsen]

How it works like!

qa.backdropcms.org is a VPS that I installed under name backdrop.vps-private.net. I think it will be proper to name it qa.backdropcms.org and use next patterns for sandboxes:

BRANCH.qa.backdropcms.org - for branches in backdrop/backdrop PR-N.qa.backdropcms.org -> for PR in backdrop/backdrop

BRANCH.contibmodulename.qa.backdropcms.org - for branches backdrop-contrib/contibmodulename PR-N.contibmodulename.qa.backdropcms.org - for PR in backdrop-contrib/contibmodulename

[quicksketch]

Thanks for the diagram, that helps a lot.

So the "webhook" code in https://github.com/backdrop-ops/github-auto is actually web hooks sent by Git.LC, not GitHub? If so, could the current code be made more clear that this is all GitLC integration, not generic webhook functionality? i.e. Rename "webhook.module" to "gitlc.module" or something? The fact that GitHub also calls them web hooks caused me confusion.

As for the current modules that are posted to that repository, do you have a preference as to whether these modules would be redistributable contrib modules? Or should we just consider them all custom modules that will only be running on backdropcms.org and nowhere else? The GitHub API module seems like we may need it as a contrib/central solution in the future, though I am fine doing things one at a time.

It's worth noting that our current GitHub integration on BackdropCMS.org (the packager) is a stand-alone module called "project_github.module" in the Project module. It has some clear overlap with the GitHub API module that you've done. If GitHub API became a contrib module, I would expect that project_github.module would depend upon it (though it doesn't have to do that necessarily). What do you think?

[Gormartsen]

I will rename webhook to gitlc_webhook module today. Also I will provide an example with contrib module(theme) sandbox.

Githubapi could be a contrib module for sure. It provide a hook integration via hook_githubapi_payload So any module (like github_issue) can provide hook implementation. I will push githubapi to contrib today as well.

[Gormartsen]

@quicksketch I checked project module right now. How do you install webhook for releases?

Do you do it manually via interface?

I am asking it, because there is a way to automate it as well.

That's exactly what I am doing for contrib sandbox example.

[quicksketch]

How do you install webhook for releases? Do you do it manually via interface?

Right now all webhooks are manually configured. You're right it could be automated via the API if you set up an access token that had the web hooks permissions. In the case of project module it's very simple because we only have to configure it in two places (once for core and once for the contrib organization). We receive two hooks, one for new releases and one for pushes to repositories (to update the readme/description on BackdropCMS.org):

The installation instructions are all in the README.md for Project GitHub. It's worth pointing out that the GitHub access token is specified in the settings.php file as $settings['project_github_access_token']. Rather than using the UI and saving to config. This is because we publicly publish our live config, and we don't want that key ending up in version control.

[Gormartsen]

Make sense.

Wow, I did not actually know that you can setup a webhook for organization!

That make everything much easier!

Will apply that for sandboxes for sure.

Basically githubapi module allow you to use "registered App".

We can have "Backdropcms.org Application" with all benefits (like registering on backdropcms.org with message "Allow backdropcms.org Application to access your public info")

[quicksketch]

Wow, I did not actually know that you can setup a webhook for organization!

Yes, just a the single web hook for all of contrib made things substantially easier. I expect we would do the same thing for sandbox testing. Just set it up once and all contrib projects would get live sandboxes on every PR.

[Gormartsen]

Mind map for PR #13 for this issue.

[quicksketch]

Okay, we've almost got this all working! We've merge all the changes in https://github.com/backdrop-ops/backdropcms.org/issues/201. New PRs are now getting automatic sandboxes!

There a couple of small issues:

• Force-pushed changes do not properly get a new sandbox with the new changes.
• We need some love for the default homepage at http://qa.backdropcms.org. Right now it's just the default landing page (in Russian) for the IT Patrol hosting company.

[klonos]

New PRs are now getting automatic sandboxes!

I just saw that! :tada: :+1: :clap:

[Gormartsen]

I added some love to http://qa.backdropcms.org

[quicksketch]

LOL, thanks @Gormartsen.

This is so cool! Check out today's PR at https://github.com/backdrop/backdrop/pull/1303. Oh yeah! So great! This requires browser testing, and we have Browser Stack accounts... it's so great to have everything working together! This is going to substantially lower the barrier to entry for end-user testing.

:tada:

[klonos]

I just tried giving a spin at the sandbox for https://github.com/backdrop/backdrop/pull/1315 over in http://1315.backdrop.backdrop.qa.backdropcms.org but instead of a fully installed Backdrop site that asks for credentials to log in, I got an installation page.

[docwilmot]

Me too.

[serundeputy]

Me three.

@jenlampton maybe push up the branch again and give backdrop-ci a chance to rebuild the sandbox?

[jenlampton]

I force-pushed. https://github.com/backdrop/backdrop/pull/1315

[quicksketch]

Let's resolve this in the follow-up made by klonos (thanks!)

A lot of the time it is working, I can't tell what's special (if anything) about the failing sandboxes. Closing and reopening seems to fix the issue some of the time.