dwylbot

"GitHub Workflow Automation, Helpful Hints & Timely Tips"

We are currently updating the documentation and the Readme of this project. All the open PRs will be reviewed soon. However if you have any questions or want to contribute to dwylbot don't hesitate to open a new issue!

Automating our GitHub workflow to .reduce the number of clicks the people need to perform
to get their work done and help everyone communicate better with team members.

Prerequisite to Understanding dwylbot

The purpose of dwylbot will remain unclear if you have not read the following:

• github.com/dwyl/github-reference
• github.com/dwyl/contributing

Please read them and come back!

Why?

dwyl's Workflow is a carefully crafted sequence of steps designed to ensure everyone in the team can communicate and see the status of work.

Learning the Workflow is never instantaneous. At the start of a project the more experienced people in the team end up having to do a bit of "workflow mentoring". e.g:

"You missed a step in the workflow which means the team is unaware of state/progress on this feature..."
"Please remember to refer to the GitHub issue in your commit messages so it is clear what each commit is for."
"Please apply the awaiting-review label when you want someone to review your work..."
"Please only assign a Pull Request for review once all the tests have passed on CI..."

We've devised a set of rules and actions to respond to these problems, they look like this:

For a list of rules we're still implementing check out: https://github.com/dwyl/dwylbot/issues/5

dwylbot automates giving "workflow related feedback" which saves everyone time!

What?

We use GitHub as our "single source of truth" (one place to keep all our information so we don't lose anything important!).
We also use GitHub to discuss questions/ideas/features/improvements, estimate the effort required to implement an idea (or "fix" an existing feature that is not working as expected) and then to track the progress while the work is being done and record how much time a person spent on the task/feature.
We refer to this as our "workflow".

dwylbot helps the humans learn & follow the workflow so we communicate better and get more done!

Install

To install and manage dwylbot on your repositories:

• visit the dwylbot home page or you can directly access the dwylbot installation page: https://github.com/apps/dwylbot

• click on "Manage dwylbot installations". This will open the dwylbot github app page.

• click configure to select the organisations where you want dwylbot to operate

• you can install dwylbot on all the repositories of the organisation or you can select specific repositories

• click install. dwylbot is now installed :tada: Just use Github as normal and dwylbot will help enhance your team's workflow.

How?

This project is written in Elixir and uses a Phoenix web server tested by Travis and running on Heroku.
If you are new to any of these tools/technologies you won't understand some of the code in this repo, so, please read/learn:

• Elixir: https://github.com/dwyl/learn-elixir
• Phoenix: https://github.com/dwyl/learn-phoenix-framework
• Travis https://github.com/dwyl/learn-travis
• Heroku: https://github.com/dwyl/heroku

Note: you don't need to be an "expert" in any of these things to start understanding the project, but it helps to know the basics.

Run The Project Locally!

You might want to run your own instance of dwylbot on your machine to see how it can help you with your workflow. You might also like to help us and add new rules to dwylbot (check our contribution guide: https://github.com/dwyl/contributing)! The "production" version of dwylbot runs on Heroku, but we develop it locally and you can easily run it on your computer and tinker the code as much as you'd like.

You will need to:

• Create a new Github application.
• Run a dwylbot server on your machine.

You'll need to have installed Elixir, Phoenix, and ngrok if you haven't already.

Note: only try to run this on your computer once you've understood Elixir & Phoenix.

Create a GitHub application

The role of the Github application is to send notifications when events occur on your repositories. For example you can get a notification when new issues or pull requests are open. In our case the Github application keep up to date dwylbot with any events happening on your repositories.

• Access the new application settings page on your Github account: Settings -> Developer settings -> Github Apps -> New Github App

  

• Github App name: The name of the app; must be unique, so can't be "dwylbot" as that's taken!

• Descriptions: A short description of the app; "My dwylbot app"

• Homepage URL: The website of the app: "https://dwyl.com/"

• User authorization callback URL: Redirect url after user authentication e.g."http://localhost:4000/auth/github/callback". This is not needed for dwylbot so this field can be left empty.

• Setup URL: Redirect the user to this url after installation, not needed for dwylbot

• Webhook URL: URL where post requests from Github are sent to. The endpoint is /event/new, however Github won't be able to send requests to http://localhost:4000/event/new as this url is only accessible by your own machine. To expose publicly your localhost server you can use ngrok. Remember to update this value after you have a running dwylbot server on your machine!

  Install ngrok. If you have homebrew, you can do this by running brew cask install ngrok

  Then in your terminal enter ngrok http 4000 to generate an SSH between your localhost:4000 and ngrok.io. Copy the ngrok url that appears in your terminal to the Github app configuration; "http://bf541ce5.ngrok.io/event/new"

  NOTE: you will need to update the webhook URL everytime you disconnect/connect to ngrok because a different URL is generated everytime.

  You can read more about webhooks and ngrok at https://developer.github.com/webhooks/configuring/

• Define the access rights for the application on the permmission section. Change "issues" and "pull requests" to "Read & Write"

• Select which events dwylbot will be notified on. Check "issues", "issue comment" and "pull request"

  Check the list of rules already implemented by dwylbot to see which permissions and notifications you want to select.

• Webhook secret: This token can be used to make sure that the requests received by dwylbot are from Github; dwylbot doesn't use this token at the moment so you can keep this field empty (see https://developer.github.com/webhooks/securing/)

• You can decide to allow other users to install the Github Application or to limit the app on your account only:

  You can now click on "Create Github App"!

• Create a private key: This key is used to identify specific dwylbot installations

  

  The downloaded file contains the private key. Copy this key in your environment variables, see the next section.

You can also read the Github guide on how to create a new Github App at https://developer.github.com/apps/building-github-apps/creating-a-github-app/

Run a dwylbot server

The dwylbot server will receive events from Github, filter and identify this events and when necessary send actions to Github (comment on issues, change labels on issues, ...)

• Clone the repository to your personal computer:

  git clone git@github.com:dwyl/dwylbot.git && cd dwylbot

• Define the local environment variables:

  If you are new to "Environment Variables", please read: github.com/dwyl/learn-environment-variables

  To run the application on your localhost (personal computer) create an .env file where you can define your environment variables.

  dwylbot/.env:

  export GITHUB_APP_ID=1111
  export GITHUB_APP_NAME=myGithubApp
  export PRIVATE_APP_KEY="-----BEGIN RSA PRIVATE KEY-----private key generated by Github when the Github app is created"
  # SECRET_KEY_BASE is required for Auth Cookie:
  export SECRET_KEY_BASE=MustBeA64ByteStringProbablyBestToGenerateUsingCryptoOrJustUseThisWithSomeRandomDigitsOnTheEnd1234567890

  You can find the value of GITHUB_APP_ID and GITHUB_APP_NAME in the About and Basic information sections. The private key is the key generated when you've crated the Gitub app. You can generate a new secrete key base with mix phoenix.gen.secret.

  Then execute the command source .env which will create your environment variables

  Note: This method only adds the environment variables locally and temporarily
  so you need to start your server in the same terminal where you ran the source command.

• Install dependencies:

  mix deps.get && npm install

• Confirm everything is working by running the tests:

  mix test

• Create the Database (if it does not already exist)

  mix ecto.create && mix ecto.migrate

• Run the Server

  mix phoenix.server

  You should see:

  [info] Running Dwylbot.Endpoint with Cowboy using http://localhost:4000

• Now that your server is running you can update the webhook url in your app config:

  • run ngrok http 4000 in a new terminal

  

  • copy and save the url into your Github App config with /event/new for endpoint

• View the Project in your Web Browser

Open http://localhost:4000 in your web browser.

From the welcome page you can now manage the installations of and select the repositories where you want dwylbot active on.

If you have managed to install successfully your new Github App on one of your repositories, you can quickly test your dwylbot server by creating for example a new issue without a description. A new dwylbot comment on the issue should warn you to add a description!

Ok, so you know the Phoenix server is working. That's nice, but what does it actually do...?

Deploy on Heroku

dwylbot is automatically deploy on Heroku each time a pull request is merged on master. If you update the version of Erlang, Elixir or Phoenix on the project, you might also need to update the buildpack on Heroku:

The elixir_buildpack.config config file allow you to update the versions of Erlang and Elixir. You can find more information about Heroku deploy here:

• https://hexdocs.pm/phoenix/heroku.html
• https://github.com/HashNuke/heroku-buildpack-elixir

Understanding The Project

Routing

Given your Phoenix knowledge, you know that the first place to look when you want to understand
a Phoenix project is: web/router.ex

In this case we only have one (interesting) route: /event/new

Make a cURL Request to the POST /event/new

Need an example GitHub Webhook request payload for this... see: https://github.com/dwyl/dwylbot/issues/6#issuecomment-286387463

See: https://developer.github.com/webhooks/creating/

tl;dr

on the team no one knows exactly what is going on at all times without having to ask.

With our GitHub-based Workflow, we successfully avoid the need for "project status update meetings":

For anyone who has never worked in a "really big" company where people have meetings about having meetings and it can feel like "there are too many steps to get work done...".
To those people we say: "you have three options:"

1. Get a job at a "Fortune 500 Company" (that has been around for 30+ years and claims to be "agile")
   .then come back chat about getting work done in teams; we will give you a shoulder to cry on!
   We promise not to say "I told you so" when you tell us we were "so right"...
2. Get a job in a small company (fewer than 10 people all co-located in the same office) where no "process" is required because they "just get stuff done!" Stay with that company long enough to feel the "growing pains" of not having a clearly defined workflow.
   .then try to retrospectively apply a workflow and teach your colleagues how to cooperate effectively.
3. Trust those of us who have felt the pain of working in (multiple) horribly complex companies and have crafted a workflow that ensures the highest level of team communication/productivity.

Organizations regularly approach us to teach dwyl's workflow to their team(s). We have done many workshops to that end.
Sadly, delivering in-person training does not scale. So we decided to automate our workflow with dwylbot.