Closed rakus closed 5 years ago
I think this sounds fine. Any feedback from @brammool? @k-takata?
I haven't used this, thus I cannot review. But so long as it's no extra work for me, and it's useful for a group of users, then why not? I suppose there will be instructions somewhere for users how to get the latest version and how to install it.
is this ready to be merged?
Christian, no. It needs cleanup (e.g. removing the commits created by Travisdaily build). Did you read the section on security? Should I work out how to do this with "Deploy Keys"?
Yeah, those commits need to be removed. if you do git rebase -i
simply delete those commits and then force push to your daily-build branch should fix it.
regarding the security: Currently, the repository already uses deploy keys, because new releases are pushed every day from my server. so that should be no problem I think.
I'm not sure we talk about the same:
"API-Token": a hexadecimal string (40 chars)
"Deploy-Keys": a encrypted SSH key
I'm not sure we talk about the same:
"API-Token": a hexadecimal string (40 chars)
"Deploy-Keys": a encrypted SSH key
I think we do. This repository uses both, an API-Token and Deploy-Keys. The API-Token is user specific and I generated it a while ago to allow travis-ci to upload the final generated appimage as Release. The Deploy-Key is an encrypted ssh-key, that is used by my server, to trigger new travis-ci builds. You see those commits here 4469b5451ffb72ba65903494195ce2cedaf84887 (subject: Vim: v<version>
). I have a crontab script, that daily runs scripts/update-repo.sh and basically updates the submodule, and commits those changes and pushes those commits back to this repository. To be able to push to this repository, I setup a dedicated ssh-key which has been generated and setup in the repository settings as deploy-key
. Once a commit is made, this triggers the build in Travis-CI, which will then build a new appimage and deploy the result back to github (using the API Token). I suppose, you have setup something similar in your branch.
It's been a while since I set it up, so I might have forgotten the gory details 😕
Ok. I think is a deploy-key
is the correct solution for commiting. This requires a different setup for the steps in the Travis phase before_deploy
.
I'll do the following:
Then I would propose to let it run again for a few days. The Cron-Build from Travis will pick up new changes from Vim and will create new commits, tags and releases.
When we are all satisfied, I will prepare the branch for merge. After it is merged, you will have to redo the Deploy-Key setup using the documentation created in step 4. (Otherwise I would know the deploy keys.)
I'm not sure how much spare time I have this week, so I might not get at it before the weekend.
Ah, okay I see where the confusion about the deploy comes from.
I'll probably won't be able to come back to this before next week anyhow, thanks!
The branch daily-build is now using a deploy key to push the new commit to GitHub. Cron job is enabled again and runs around 07:00 CEST.
Lets monitor this for a few days before merging. Then I will prepare the branch for merging again (removing created commits, rebase master).
This is what you have to do then:
1) Merge the PR
2) Checkout master
3) Delete the file scripts/github_deploy_key.enc
Then you need to create your own deploy key. I followed the description here from section 1 to 2.5.
Differences to the description:
scripts
, not in the root dir of the project. Change to scripts
before executing ssh-keygen ...
etc.When you encrypt the key with travis encrypt-file github_deploy_key
it will print a line
openssl aes-256-cbc -K $encrypted_XXXXXXXXXXXX_key -iv $encrypted_XXXXXXXXXXXX_iv -in github_deploy_key.enc -out github_deploy_key -d
This line has to replace the similar line in .travis.yml
in section before_deploy
. Note that you have to add scripts/
before github_deploy_key.enc
.
When you finally push the changes back to Github, this will trigger a build on travis. Most likely this will not commit and push anything, except there are some new commits to the Vim repository.
Configure the daily build:
Go to Travis, select the project "vim-appimage". Open the settings page ("More options" -> "Settings"). The Cron Job settings are located at the bottom.
Here comes a big disadvantage: You can only set the cron job to daily, but the time is always the time you created this cron job. If you configure the cron job at 7 in the morning, it will always run around that time. If you want the Job to run at 02:00 at night, then you have to configure it at 02:00 :-(. Or maybe use some browser automation.
BTW: You might wonder about two sleep 10
in the travis file. I had error situations where the build failed (coding error), but the failing command was not in the logs. So I added those two sleeps, to make sure that all output is in the travis logs. Background is that Travis filters all output to make sure no secure variables (like $encrypted_XXXXXXXXXXXX_key
) are printed to the logs.
thanks for the detailed explanation. I think I have done everything like you wanted. (I merged it locally, re-edited the your last-commit and replaced those values with the new values from the locally generated keyfile.)
Hopefully everything works now as expected. I guess, we'll see ;)
That was quicker than I expected. I wanted to do a final cleanup in the section deploy
.
Did you see the variable $API_TOKEN
in the section deploy
? This is a github API token.
You need to create a secure Environment Variable in Travis for that. See here.
Alternative would be to use the encrypted API-token, as it was used before:
deploy:
provider: releases
api-key:
secure: KMER+Njo7lI....
Also you can remove the lines
on:
tags: true
Then it will only deploy from the master branch (tagged or not) and will always upload the AppImage. The later part might be useful if you test new configure options etc.
ah, good hint. I did indeed forget about that change. I reverted to the old secure: KMER
notation, so that the script does not depend on a specific environment variable that is set in another web application. Now it only depends on my personal security token, which is also not ideal, but hopefully I remember next time ;)
Thanks again.
@rakus
something did not quite work. The name of the deployment is only Vim:
it should probably be Vim: version
and this is how your repository looks like. I think it should be $TRAVIS_TAG and that variable must have been defined according to the commit to this repository, so I am wondering what is missing... 😕
Damned.
We had two builds.
The first one was OK
The second one created the wrong name for the release.
TRAVIS_TAG
was not setTRAVIS_TAG
was unsetTRAVIS_TAG
Conclusion: We need the on-tags: true
in the deploy section.
My last explanation why it could be dropped was not that bright ;-).
I would propose the following untested change to .travis.yml
:
diff --git a/.travis.yml b/.travis.yml
index 4a0c4cc..4b435a3 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -62,7 +62,7 @@ before_deploy:
- sleep 10
# Set up git user name and tag this commit
- |
- if [ -n "$TRAVIS_TAG" ]; then
+ if [ -n "$TRAVIS_TAG" ] && [ "$TRAVIS_BRANCH" = "master" ]; then
echo "Committing, Tagging and Pushing..."
openssl aes-256-cbc -K $encrypted_62bc4630de43_key -iv $encrypted_62bc4630de43_iv -in scripts/github_deploy_key.enc -out github_deploy_key -d
chmod 600 github_deploy_key
@@ -78,6 +78,7 @@ before_deploy:
rm -f github_deploy_key
else
echo "Nothing to commit"
+ unset TRAVIS_TAG
fi
- RELEASE_BODY="$(cat $TRAVIS_BUILD_DIR/release.body)"
@@ -92,6 +93,8 @@ deploy:
overwrite: true
body: "$RELEASE_BODY"
skip_cleanup: true
+ on:
+ tags: true
after_script:
# workaround for truncated output
This:
on-tags:true
TRAVIS_TAG
in before_deploy
when not on master branch, so no deployment from other branches.okay, I included your patch
This PR is opened as a draft, to start a discussion about this.
This push request adds the ability to run unattended daily builds that would automatically update to the latest Vim release via Travis cron jobs.
The currently working state can be monitored in my fork of vim-appimage on the branch daily-build. The daily build runs via Travis cron job at ~ 13:20 UTC.
On 2019-04-30 the build released this version. While doing the build it fetched the newest Vim changes and commited them to github with this commit.
This is not finished yet, see section Security. I will only continue, if you are interested.
New Build Sequence
before_install
VIM_SUMMARY
(git submodule summary vim
)VIM_SUMMARY
is not empty (aka there are new commits in the submodule), store the latest tag of the Vim submodule asTRAVIS_TAG
.script
before_deploy
If the
TRAVIS_TAG
is not set, do nothing.If the
TRAVIS_TAG
is set: Commit the changes in the submodulevim
and build the commit message fromTRAVIS_TAG
,VIM_SUMMARY
, theTRAVIS_BUILD_WEB_URL
and the string[ci skip]
. The[ci skip]
is needed to prevent instant rebuild after the push. Then tag the current state with theTRAVIS_TAG
. Finally push the commit and tag to GitHub.deploy
Runs as before.
Security
As this commits and pushes changes to the git repository, special care has to be taken.
There are two ways.
Personal Access Token
Found in GitHub under "User Settings" -> "Developer Tools" -> "Personal Access Token".
The created token has to have write access to public git repositories.
Advantage: Easy
Disadvantage: Not repository specific, valid for all repositories of user.
This is at least needed to upload the AppImage to Github.
Deploy Keys
Found in GitHub under "Project Settings" -> "Deploy Keys".
This works with encrypted SSH keys and is repository specific.
Advantage: Repository specific
Disadvantage: Needs a few steps to setup.
Decision
Currently I use "Personal Access Token" to authorize the commits, as it is a no-brainer.
I assume "Deploy Keys" is the better way, but I need to evaluate and test it.
... to be done