Daily build - Githubissues

rakus commented 5 years ago

This PR is opened as a draft, to start a discussion about this.

This push request adds the ability to run unattended daily builds that would automatically update to the latest Vim release via Travis cron jobs.

The currently working state can be monitored in my fork of vim-appimage on the branch daily-build. The daily build runs via Travis cron job at ~ 13:20 UTC.

On 2019-04-30 the build released this version. While doing the build it fetched the newest Vim changes and commited them to github with this commit.

This is not finished yet, see section Security. I will only continue, if you are interested.

New Build Sequence

before_install

Checkout current branch, to make sure we are not in some disconnected HEAD state.
In the vim submodule checkout master and pull the latest changes
Store the list of new commits in VIM_SUMMARY (git submodule summary vim)
If VIM_SUMMARY is not empty (aka there are new commits in the submodule), store the latest tag of the Vim submodule as TRAVIS_TAG.

script

builds Vim
builds the AppImage
run the Vim tests with the AppImage

before_deploy

If the TRAVIS_TAG is not set, do nothing.

If the TRAVIS_TAG is set: Commit the changes in the submodule vim and build the commit message from TRAVIS_TAG, VIM_SUMMARY, the TRAVIS_BUILD_WEB_URL and the string [ci skip]. The [ci skip] is needed to prevent instant rebuild after the push. Then tag the current state with the TRAVIS_TAG. Finally push the commit and tag to GitHub.

deploy

Runs as before.

Security

As this commits and pushes changes to the git repository, special care has to be taken.

There are two ways.

Personal Access Token

Found in GitHub under "User Settings" -> "Developer Tools" -> "Personal Access Token".

The created token has to have write access to public git repositories.

Advantage: Easy

Disadvantage: Not repository specific, valid for all repositories of user.

This is at least needed to upload the AppImage to Github.

Deploy Keys

Found in GitHub under "Project Settings" -> "Deploy Keys".

This works with encrypted SSH keys and is repository specific.

Advantage: Repository specific

Disadvantage: Needs a few steps to setup.

Decision

Currently I use "Personal Access Token" to authorize the commits, as it is a no-brainer.

I assume "Deploy Keys" is the better way, but I need to evaluate and test it.

... to be done

chrisbra commented 5 years ago

I think this sounds fine. Any feedback from @brammool? @k-takata?

brammool commented 5 years ago

I haven't used this, thus I cannot review. But so long as it's no extra work for me, and it's useful for a group of users, then why not? I suppose there will be instructions somewhere for users how to get the latest version and how to install it.

chrisbra commented 5 years ago

is this ready to be merged?

rakus commented 5 years ago

Christian, no. It needs cleanup (e.g. removing the commits created by Travisdaily build). Did you read the section on security? Should I work out how to do this with "Deploy Keys"?

chrisbra commented 5 years ago

Yeah, those commits need to be removed. if you do git rebase -i simply delete those commits and then force push to your daily-build branch should fix it.

regarding the security: Currently, the repository already uses deploy keys, because new releases are pushed every day from my server. so that should be no problem I think.

rakus commented 5 years ago

I'm not sure we talk about the same:

"API-Token": a hexadecimal string (40 chars)

"Deploy-Keys": a encrypted SSH key

chrisbra commented 5 years ago

I'm not sure we talk about the same:

"API-Token": a hexadecimal string (40 chars)

"Deploy-Keys": a encrypted SSH key

I think we do. This repository uses both, an API-Token and Deploy-Keys. The API-Token is user specific and I generated it a while ago to allow travis-ci to upload the final generated appimage as Release. The Deploy-Key is an encrypted ssh-key, that is used by my server, to trigger new travis-ci builds. You see those commits here 4469b5451ffb72ba65903494195ce2cedaf84887 (subject: Vim: v<version>). I have a crontab script, that daily runs scripts/update-repo.sh and basically updates the submodule, and commits those changes and pushes those commits back to this repository. To be able to push to this repository, I setup a dedicated ssh-key which has been generated and setup in the repository settings as deploy-key. Once a commit is made, this triggers the build in Travis-CI, which will then build a new appimage and deploy the result back to github (using the API Token). I suppose, you have setup something similar in your branch.

It's been a while since I set it up, so I might have forgotten the gory details 😕

rakus commented 5 years ago

Ok. I think is a deploy-key is the correct solution for commiting. This requires a different setup for the steps in the Travis phase before_deploy.

I'll do the following:

Stop the cron job
Revert all the commits created by Travis (v8.1.1239 - v8.1.1275)
Rebase the branch on the current master from vim/vim-appimage
Evaluate Deploy-Keys and write a doc how to set it up with Travis.
Implement Deploy-Keys for the branch "daily-build"
Reenable cron job

Then I would propose to let it run again for a few days. The Cron-Build from Travis will pick up new changes from Vim and will create new commits, tags and releases.

When we are all satisfied, I will prepare the branch for merge. After it is merged, you will have to redo the Deploy-Key setup using the documentation created in step 4. (Otherwise I would know the deploy keys.)

I'm not sure how much spare time I have this week, so I might not get at it before the weekend.

chrisbra commented 5 years ago

Ah, okay I see where the confusion about the deploy comes from.

I'll probably won't be able to come back to this before next week anyhow, thanks!

rakus commented 5 years ago

The branch daily-build is now using a deploy key to push the new commit to GitHub. Cron job is enabled again and runs around 07:00 CEST.

Lets monitor this for a few days before merging. Then I will prepare the branch for merging again (removing created commits, rebase master).

This is what you have to do then:

1) Merge the PR 2) Checkout master 3) Delete the file scripts/github_deploy_key.enc

Then you need to create your own deploy key. I followed the description here from section 1 to 2.5.

Differences to the description:

the encrypted key is located in the directory scripts, not in the root dir of the project. Change to scripts before executing ssh-keygen ... etc.
The ssh-key is only decrypted if a commit and push is required.
The ssh-key is deleted immediately after usage.

When you encrypt the key with travis encrypt-file github_deploy_key it will print a line

openssl aes-256-cbc -K $encrypted_XXXXXXXXXXXX_key -iv $encrypted_XXXXXXXXXXXX_iv -in github_deploy_key.enc -out github_deploy_key -d

This line has to replace the similar line in .travis.yml in section before_deploy. Note that you have to add scripts/ before github_deploy_key.enc.

When you finally push the changes back to Github, this will trigger a build on travis. Most likely this will not commit and push anything, except there are some new commits to the Vim repository.

Configure the daily build:

Go to Travis, select the project "vim-appimage". Open the settings page ("More options" -> "Settings"). The Cron Job settings are located at the bottom.

Here comes a big disadvantage: You can only set the cron job to daily, but the time is always the time you created this cron job. If you configure the cron job at 7 in the morning, it will always run around that time. If you want the Job to run at 02:00 at night, then you have to configure it at 02:00 :-(. Or maybe use some browser automation.

BTW: You might wonder about two sleep 10 in the travis file. I had error situations where the build failed (coding error), but the failing command was not in the logs. So I added those two sleeps, to make sure that all output is in the travis logs. Background is that Travis filters all output to make sure no secure variables (like $encrypted_XXXXXXXXXXXX_key) are printed to the logs.

chrisbra commented 5 years ago

thanks for the detailed explanation. I think I have done everything like you wanted. (I merged it locally, re-edited the your last-commit and replaced those values with the new values from the locally generated keyfile.)

Hopefully everything works now as expected. I guess, we'll see ;)

rakus commented 5 years ago

That was quicker than I expected. I wanted to do a final cleanup in the section deploy.

Did you see the variable $API_TOKEN in the section deploy? This is a github API token. You need to create a secure Environment Variable in Travis for that. See here.

Alternative would be to use the encrypted API-token, as it was used before:

deploy:
  provider: releases
  api-key:
    secure: KMER+Njo7lI....

Also you can remove the lines

on:
      tags: true

Then it will only deploy from the master branch (tagged or not) and will always upload the AppImage. The later part might be useful if you test new configure options etc.

chrisbra commented 5 years ago

ah, good hint. I did indeed forget about that change. I reverted to the old secure: KMER notation, so that the script does not depend on a specific environment variable that is set in another web application. Now it only depends on my personal security token, which is also not ideal, but hopefully I remember next time ;)

Thanks again.

chrisbra commented 5 years ago

@rakus something did not quite work. The name of the deployment is only Vim: it should probably be Vim: version and this is how your repository looks like. I think it should be $TRAVIS_TAG and that variable must have been defined according to the commit to this repository, so I am wondering what is missing... 😕

rakus commented 5 years ago

Damned.

We had two builds.

The first one was OK

It pulled the new patch 1330 from vim repo
commited, tagged & pushed
created the new release Vim: v8.1.1330 (I'm pretty sure I saw it yesterday)

The second one created the wrong name for the release.

pull did not get anything new from vim repo, so TRAVIS_TAG was not set
did not commit/push, as TRAVIS_TAG was unset
Uploaded the AppImage to the right tag (via commit matching), but the title misses the TRAVIS_TAG

Conclusion: We need the on-tags: true in the deploy section. My last explanation why it could be dropped was not that bright ;-).

I would propose the following untested change to .travis.yml:

diff --git a/.travis.yml b/.travis.yml
index 4a0c4cc..4b435a3 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -62,7 +62,7 @@ before_deploy:
   - sleep 10
   # Set up git user name and tag this commit
   - |
-    if [ -n "$TRAVIS_TAG" ]; then
+    if [ -n "$TRAVIS_TAG" ] && [ "$TRAVIS_BRANCH" = "master" ]; then
       echo "Committing, Tagging and Pushing..."
       openssl aes-256-cbc -K $encrypted_62bc4630de43_key -iv $encrypted_62bc4630de43_iv -in scripts/github_deploy_key.enc -out github_deploy_key -d
       chmod 600 github_deploy_key
@@ -78,6 +78,7 @@ before_deploy:
       rm -f github_deploy_key
     else
       echo "Nothing to commit"
+      unset TRAVIS_TAG
     fi
   - RELEASE_BODY="$(cat $TRAVIS_BUILD_DIR/release.body)"

@@ -92,6 +93,8 @@ deploy:
     overwrite: true
     body: "$RELEASE_BODY"
     skip_cleanup: true
+    on:
+      tags: true

 after_script:
   # workaround for truncated output

This:

Re-introduces the on-tags:true
Only commits/tags/pushes on the master branch
Unsets the TRAVIS_TAG in before_deploy when not on master branch, so no deployment from other branches.

chrisbra commented 5 years ago

okay, I included your patch

vim / vim-appimage

Daily build #7

New Build Sequence

Security

Personal Access Token

Deploy Keys

Decision