Reliability monitoring - e2e testing of GitHub workflow

alexellis commented 5 years ago

Context

Since moving the Communtity Cluster from GKE to DigitalOcean the reliability has dropped dramatically due to issues with the Cilium networking driver 1.4.0. A separate action will be required for this, but in the meantime it highlights a lack of e2e monitoring for the GitHub workflow end-to-end.

Task

Create a Golang function that will:

clone the repo (it will be a Go function)
modify the handler's response to include a specific string such as the current date/time stamp
do a git push (using a secret)
call the function in a loop with a generous timeout until the response contains the new string
call into a slack webhook if the string is incorrect or not 200 at the end

Constraints

The repo in question should be cloned with the go-github library
This should be an OpenFaaS function
Use /tmp/ as a writable area
For write access use the GitHub App approach as per alexellis/derek and openfaas/openfaas-cloud

Suggestions and input is welcome

matipan commented 5 years ago

The function that will be used to test could simply return the contents of an environment variable that it's set at deployment time on the stack.yml file. That way we only modify the yaml and not the actual code of the function. WDYT?

alexellis commented 5 years ago

I was thinking that changing the code would be part of the e2e test to exercise the container builder. Changing an environmental variable is probably also worth doing separately in the same test to prove deployments are being updated.

matipan commented 5 years ago

You are right. We'll do both then :+1:

matipan commented 5 years ago

Should this functions use the new of-watchdog with the golang-http template?

alexellis commented 5 years ago

I think that could work 👍

matipan commented 5 years ago

I seems that the github API does not support cloning or pushing to repositories since that uses the git protocol(related issue). So we'll need to use plain git to clone the repo. This means that the golang-http template cannot be used since we need to have git installed in the final image. I'll use the dockerfile template instead.

alexellis commented 5 years ago

What about https://developer.github.com/v3/repos/contents/ ?

matipan commented 5 years ago

My bad for not digging a bit more. Thanks alex! :+1:

matipan commented 5 years ago

I have both functions implemented but I encountered a few issues along the way.

Authentication as application: in order to modify the repository that the function will use for testing, the application has to be authenticated as an installation(we don't have write-access otherwise). This means that once we(or whoever uses this function to test their ofc installation) install the github app we need to grab the installation number and add it to the environment. I don't think this is an issue but I thought it was worth to mention, maybe there is another way?
Pushing as an application: for some reason I was unable to trigger a build after the github app made a push. The only difference in the push events I found between a regular user and the github app is that the pusher field for the app had the email set to null. Are we ignoring those in ofc? I implemented the function using a personal access token to move along the development but we need to figure out why this is happening. Maybe we could look into the OpenFaaS Cloud github app events and see what it responded or why it didn't pick up that event?
Timeouts: we need to document that this function can take more than 60s to finish executing. I had to modify the timeouts of my nginx-ingress controller since the default was 60s and the function was taking longer than that.

I added a bit of logging to the function:

2019/02/25 22:10:20 stdout: Rate limiting github.Rate{Limit:5000, Remaining:4999, Reset:github.Timestamp{2019-02-25 23:10:20 +0000 UTC}}
2019/02/25 22:10:21 stdout: Rate limiting github.Rate{Limit:5000, Remaining:4998, Reset:github.Timestamp{2019-02-25 23:10:20 +0000 UTC}}
2019/02/25 22:10:52 stdout: Attempt: 0
2019/02/25 22:10:52 stdout: Function responded with: 2019-02-25 20:34:32.879175321 +0000 UTC m=+586.922022707 - Expected: 2019-02-25 22:10:20.66471465 +0000 UTC m=+27.460919558
2019/02/25 22:10:52 stdout: Attempt: 1
2019/02/25 22:11:22 POST / - 200 OK - ContentLength: 4
2019/02/25 22:11:22 stdout: Function responded with: 2019-02-25 22:10:20.66471465 +0000 UTC m=+27.460919558 - Expected: 2019-02-25 22:10:20.66471465 +0000 UTC m=+27.460919558

Whenever the function finishes the output is a simple true or false, if we want more info I can add it: e2e-test

alexellis commented 5 years ago

Hi @matipan this looks like a very good summary of your learnings. I'll get back to you with some thoughts soon.

Do you have the repo available for me to look at the differences in the commits?

Maybe we could look into the OpenFaaS Cloud github app events and see what it responded or why it didn't pick up that event?

If you have your own OFC GitHub App did the commit appear differently on the "GitHub App" request/response page (use the Advanced tab)? For me to get the event we'd need to do it live on Slack so that I can find the event.

Alex

matipan commented 5 years ago

Sure! This is a commit pushed by the github app although I used my information as Committer and Author. And this is a commit pushed using the personal access token

I'm using the OpenFaaS Cloud Community Github App. But I installed my own app(that basically did nothing) that was subscribed to the same events the github app was subscribed, I'll send over slack the payloads that github sent for the different commits :+1:

matipan commented 5 years ago

This is the code I've written for the test so far: https://github.com/matipan/ofc-e2e-test

matipan commented 5 years ago

@alexellis Good news! I found the reason why it would not work with a github app. Basically it's because the sender of the event is the github app, not the user, and the github app was not in the CUSTOMERS file. If I add it to the file it works ok :+1: Important detail: we need to increase the proxy-read-timeout of openfaas's ingress since by default it's only 60 seconds and is not enough to run the entire test.

matipan commented 5 years ago

I'm going to cleanup the code a bit and have it ready for review tonight/tomorrow

matipan commented 5 years ago

I have the test ready and working on my own ofc setup: e2e-test I was thinking I could do a PR to openfaas-cloud and add the ofc-test function as a separate folder. We still need to create a new repo for the hello-world function that will be used to test. What do you think @alexellis ?

openfaas / openfaas-cloud