deployment variable in a (secure) environment variable

peterjc commented 5 years ago

Unfinished attempt to implement #348, adds a new command line option --dkenv short for deploy key in environment variable.

As far as I can tell, doctr configure --dbenv DOC_KEY ... works nicely. Building on the existing code this generates a key, sends the public key to the destination GitHub repository, and prints the private key to the terminal with instructions to add it to the source repository's TravisCI config.

However, something is breaking on the doctr deploy --dbenv DOC_KEY ... side, possibly in how I have been attempting to test is locally, and I don't understand enough of the design to immediately see where.

e.g. Something like this with --force and setting the $TRAVIS_REPO_SLUG variable to attempt to run locally:

$ TRAVIS_REPO_SLUG=owner/source doctr deploy ${TRAVIS_TAG:-dev} --dkenv DOC_KEY --build-tags --built-docs Doc/api/_build/html/ --deploy-repo owner/dest --force

asmeurer commented 5 years ago

I wouldn't try to test locally. I can give you push access to push this up directly and you can test it on Travis. I'm not clear if that gives you access to upload deploy keys so if it doesn't just let me know and I'll upload it.

peterjc commented 5 years ago

Looks like some useful error from TravisCI as is.

Trying the deploy stuff as a shell script (locally), turned up another issue. The branch is here in the short term:

https://github.com/peterjc/biopython/tree/deploy

Called this file .github/ssh_via_deploy_key.sh:

#!/bin/bash
# Call ssh using our GitHub repository deploy key (set via -i)
# using -F to make sure this ignores ~/.ssh/config
ssh -i "$HOME/.biopython_doc_deploy.key" -F /dev/null -p 22 $*

Called by:

#!/bin/bash

set -euo pipefail

# Assumes being called from the Biopython repository's root folder,
# (i.e. a clone of https://github.com/biopython/biopython) as part
# of our continuous integration testing to save the compiled docs
# to https://github.com/biopython/docs
#
# In order to have write permissions, we put a private key into the
# TravisCI settings as a secure environment variable, and put the
# matching public key into the GitHub documentation repository's
# settings as a deploy key with write permissions.

DEST_SLUG=biopython/docs
DEST_DIR=${TRAVIS_TAG:-dev}
SOURCE_DIR=${TRAVIS_BUILD_DIR:-.}/Doc/api/_build/html
WORKING_DIR=/tmp/deploy_biopython_docs

# This will make git use our deployment key,
# using sed to restore line breaks 
echo $DOC_KEY | sed -E -e 's/ RSA PRIVATE /private/g' | sed -E -e 's/[[:space:]]+/\n/g' | sed -E -e 's/private/ RSA PRIVATE /g' > $HOME/.biopython_doc_deploy.key
chmod 600 $HOME/.biopython_doc_deploy.key
export GIT_SSH=${TRAVIS_BUILD_DIR:-$PWD}/.github/ssh_via_deploy_key.sh

# Clone the destination under /tmp (public URL, no key needed)
rm -rf $WORKING_DIR
git clone https://github.com/$DEST_SLUG.git $WORKING_DIR

# Update the files (assumes nothing removed)
mkdir -p $WORKING_DIR/$DEST_DIR
cp -R $SOURCE_DIR/* $WORKING_DIR/$DEST_DIR/

pushd $WORKING_DIR
git checkout gh-pages
# Switch the git protocol to SSH based so we can use our key
git remote set-url origin --push git@github.com:$DEST_SLUG.git
git add $DEST_DIR/
git commit -m "Automated update" --author "TravisCI <travisci@example.org>"
git push origin gh-pages
popd

echo "Documentation deployed!"

The key finding from these experiments is a little massaging of the public key is needed to restore the line lines - done in the above with sed, but would be clearer in Python as part of doctr.

peterjc commented 5 years ago

Another important lesson from trying this on TravisCI,

Setting environment variables from repository settings
$ export DOC_KEY=[secure]
We were unable to parse one of your secure environment variables.
Please make sure to escape special characters such as ' ' (white space) and $ (dollar symbol) with \ (backslash) .
For example, thi$isanexample would be typed as thi\$isanexample. See https://docs.travis-ci.com/user/encryption-keys.

Most likely we need to print the private key to screen escaped suitably for TravisCI to accept it (spaces and new lines), and unescape in the script.

asmeurer commented 5 years ago

Can we base64 encode it?

asmeurer commented 5 years ago

Actually the private key should already be base64 encoded. There's probably an issue with the header part of the file. This is what a private key that I just generated with ssh-keygen looks like

So we should remove the -----BEGIN OPENSSH PRIVATE KEY----- parts and any newlines, then when writing it back to a file in Travis, add those back in. I don't know how picky it is about having it in that exact format.

asmeurer commented 5 years ago

I forgot that we use cryptography now to generate the ssh key instead of ssh-keygen. This is what it looks like

>>> doctr.local.generate_ssh_key()
(b'-----BEGIN PRIVATE KEY-----\nMIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQC6rPXZjhn82O3+\nvWBF0mZocRTKAdOxRvdl8ezF+P2B7+tjr0uoMveTFDSLny6dIf9j+NLGAmCywtT6\n3+/KOBZXiMFIfACcepQt9CEUWJfPZSo6/u8cjGneuZHNiT9z9Jo/PbZDAeKPGbdq\nZgTtJxb01Lo51zae5Z+rqiW5HqAwK2wpV/uz8FGvcsRN4luTjrgagG8o8Eu4AMzP\nZo0NiwSn+jY9SyrJ4ynMUXdA4AyvulYPMzDZtayNyws+31M5PQYMWmuVbdJ2wSnz\nfyvqchUay0jAnXytJ87c4wKbjrMeaAuy1QfOkpw/vMXQn8xvAepIrXvbwA+6tGne\nldXnXTQgPNtaYZC5vc0PRZAAbsAHf/5qQjk1mut9v6meYUnRZgKE8qWp0Jwo6JFi\nXWQ+9+Ba/WcqqVwtAho7vydvxKM72oMVLRhg3psYWJduQfa+p6vs6DJUBBf4cTIA\nzXg+ZJ0jZJ+UX5gmitI4p2gdstzQX95SbvLCPwxElAAOnmH5rKPR32JK08RmtwBc\nPDabQs/u+DY1EC97nb2VjM4QOH6xfc7pyYI1d0x+rPqRh8TZrZCTnLa5i8F9lMVn\nLColU0KzUdRzuBHaTFeUOuVnpPJ7j/VEmA8hFP03dCLHRt36tI++xW89Gd+I7seX\nfl1u1k0KxBuI8EVyTF8sPTz/vjS3XQIDAQABAoICAQCrlyrRVHZ83aY+fzLMW284\n16EFYpKFJLdqJOvAunPECZX0ZoCD1n2N24fFQ4fkdgi8i06rJipehwqgpFFVLyMS\nSSlpiFpDe8VTTFFP03OP9uzPl+CQ/FyglzD3ng4OdyuFsCMmCsiHQ1s+WRJ0L3dB\ni3y2iCWz/1w6vka4l/ck7/UXN8GtD9z2Ced5s/T7eLev3JjRJ7hiJZIdnqVPapbY\nFP3gb4SgWMfmAIg+wPPIX96VUDe6Fu3K1HW80Ck+tuIlXsP/chiAgmQeZ6olccIG\nhA+WxeyBedMDZUPTW2M4Mul1862ea1Nmnw2yDAEtlLQXJChywWNz+jxKlq4tYpXy\n7jgg9V6wcI5FkduRIkhVPny1ZZw1kMqycl0R8VzkR0vEBioC8WVrZhkHAEswWEhS\nhtmQ2xgstlPDkHSImpZCqWOS3v0gDxv2+9Sw27nSAi9zNsZKcfp4roDvPK3+9l2m\nOR17u1Vj9OQpMsMQ7OQy9ZH4NIQEI3DV+nf2JS9b8wFhAOhgLgnMnF5Frdpmxtcm\nU0n+gXemEl+13k8FBx7Ub4beRC5Xo++AifOs+O5GSPS9fhpMmIWHIk/e0ZATABch\nOf0VhYpamj86rxhLCQwjkvMO8EqDqFBwsY4XZVT8VilNU/M58BLKgKWxODGFj6ED\n+SUrVY+N8Nz/i8+2jDrhPQKCAQEA2/jFKpZgl99J8KkjBfhXoBxF4HSPUrIQg4s1\nK0Ewg0PMNLvAbDfXV1xxJ+xLyXNaWQwELh3oIdAe8wt2ftcSS1V4xHemf7oysBB2\n4cd2lHJy/+TjOzjocmo2cMlfp0aRKFvgt0B67gNhW6hYCas8TpKQBrVaHRdBHs+m\n5+rVkX5KQzafIYiRJ9tBmKUwO3Rk2HGHYovyuDY4C+1sNfXE5MyIYaRLQLAkvDa4\nnj9Rq6z6fI5Yb2gwYFJ6/5nT9IchjwmEzGKqGeeG761Xp7v+tJTZ1ka37AVl0qTW\nHMOnAxgQm1AJdcdG4oyEjFkWbdDD4fXq7nTuQ1azVFE4J2E2zwKCAQEA2UAci5oF\n/shgyjUGBOMSzCTpi3z2W3cjajHqUyK6DREKC6Jrldjec7EIx68DdI7th+VnWlwW\n2/7RjqfKGxkfGj7IUUFMgTlBHEq4mVFRVKhqHFhdh1FBPnH+ATn8hgOp0CxGP4v/\nqO1lLXLHe/Kar2cK8zNlbHg5/u1i6PuC3K0WbM+8CdXDvn+qwl5oUQQjKs/GZoVp\nS+cKJtiT50DH2qoOx9U6vwsguNOgWyEZgQ3Miss0NdMb+D/FBDXD2WjWPQkJ+ruR\nFdWZwR/Jx7JpboDlrNJpCZsIevunYqnR+23U+ZxBTnH9dAJkuIrSFhMm6+YviSOg\nk/uBSnX/C/F6EwKCAQEAz8f37hdnnG2VeVc6tvvzQVETjEZtz25VfPv0uCv2uDdF\nYBZtV4uTxHiUhmKE4AAvOmfIVwt25uGhKnEMeBmNtU1CK0reIk5ubLLQqMpxrx1A\nlYjOP3Ws086SKA1/ZhGZMec/p7mnpMXao+qrZk6yQ4HbvAp32XzKzWDWRsEjBTCm\n00B4JgPLITvRhW+b1L1IOM9cU/Dfz7OfU1zsVzgUyQ6OULURREReHs8NqqUi7ygQ\n37DRxkJDV+jxOBlFBfjS8TrLjwgvpxJ0+lbhspY4rLjh366jMrWSjduYSEljq9+C\naEK8/NzEj2CuH6hTMF3/eaSCSsZ2/XKbKC0j/sasLwKCAQEAgidtspkpJFYp3prb\nq0vbNCCdJmtMMMn0lqem6f2xFyjxKr041UJjK06RowgP+uGyHqtqOvFW5KAKLfwK\nEif/wTqBymRjkDub7XY6l+fm4OAxCiBKkEo221FxyoxR5HwHXWdZArM+DJeE+TB9\noJ1c3N7P6ZoOFmkE3dycWFZuNQUhnTjrP70ok1VrGR10Q61F4F0wULV2uvmE1HcG\nTRI7aZ5eUoxFsLTa+sAWnuH6pJ1+wFwzQFfkttqFjxsi5Xpwd4qVxvheWIVqoxAH\nVDNoBMMGVn6MXSvbbcqconh5C7fmU1Cws22JWdohO4o3iPAablOugOuuRVn1QIXm\nseIOrwKCAQBlMAi+1WJI3aEqW3Ej3EcfP6Vp2Y1lu/p4Wj7PLoExxV9iRudU6RxQ\ndNX8JvubJUxaXGgSMgEFT5rV5q8G8Ig7tVHG0WjgTeiW9JN+i9woWygybMl7Kcyz\nAUVDQFxsC/bLAvk0bZ+PuFpmTG+rzIeEE1o8RqF9MOnffqeD0thfdt20djC4aLdp\nkI0Wl9FGpCmuLiioXC40WOLnB1sD7ykRr6VKzPxyukESgd1WQ97SlM7ol47idTKS\n+NmfeIj1zbp5cfaAq+uwh0HZ0lYJ7Jub44BJov+HLBBScTivKUIhRrkbIyzudFwZ\nmLoKXnxd/xtDY5UfWxNqpc2iVsdyYgqf\n-----END PRIVATE KEY-----\n', b'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC6rPXZjhn82O3+vWBF0mZocRTKAdOxRvdl8ezF+P2B7+tjr0uoMveTFDSLny6dIf9j+NLGAmCywtT63+/KOBZXiMFIfACcepQt9CEUWJfPZSo6/u8cjGneuZHNiT9z9Jo/PbZDAeKPGbdqZgTtJxb01Lo51zae5Z+rqiW5HqAwK2wpV/uz8FGvcsRN4luTjrgagG8o8Eu4AMzPZo0NiwSn+jY9SyrJ4ynMUXdA4AyvulYPMzDZtayNyws+31M5PQYMWmuVbdJ2wSnzfyvqchUay0jAnXytJ87c4wKbjrMeaAuy1QfOkpw/vMXQn8xvAepIrXvbwA+6tGneldXnXTQgPNtaYZC5vc0PRZAAbsAHf/5qQjk1mut9v6meYUnRZgKE8qWp0Jwo6JFiXWQ+9+Ba/WcqqVwtAho7vydvxKM72oMVLRhg3psYWJduQfa+p6vs6DJUBBf4cTIAzXg+ZJ0jZJ+UX5gmitI4p2gdstzQX95SbvLCPwxElAAOnmH5rKPR32JK08RmtwBcPDabQs/u+DY1EC97nb2VjM4QOH6xfc7pyYI1d0x+rPqRh8TZrZCTnLa5i8F9lMVnLColU0KzUdRzuBHaTFeUOuVnpPJ7j/VEmA8hFP03dCLHRt36tI++xW89Gd+I7seXfl1u1k0KxBuI8EVyTF8sPTz/vjS3XQ==')

asmeurer commented 5 years ago

Or alternately we can change the encoding argument in generate_ssh_key https://cryptography.io/en/latest/hazmat/primitives/asymmetric/serialization/#serialization-encodings. Then likely use another cryptography function to go from that to a valid id_rsa file.

peterjc commented 5 years ago

Right now I have been using this in bash to recover the key file:

# This will make git use our deployment key, using sed to restore line breaks.
# On TravisCI, seems must create the variable using '\ ' and '\n'
# The sed commands unescape these (but the new line one fails on macOS):
echo $DOC_KEY | sed "s/\\\ / /g" | sed 's/\\n/\n/g' > $HOME/.biopython_doc_deploy.key
chmod 600 $HOME/.biopython_doc_deploy.key
export GIT_SSH=${TRAVIS_BUILD_DIR:-$PWD}/.github/ssh_via_deploy_key.sh

Note it won't necessarily be -----BEGIN PRIVATE KEY-----, depending how the key-pair is made it could be something like -----BEGIN RSA PRIVATE KEY----- instead (with a key I generated locally). This seems to cover the details:

https://stackoverflow.com/questions/20065304/differences-between-begin-rsa-private-key-and-begin-private-key

asmeurer commented 5 years ago

The SO answer says PKCS#8 uses "BEGIN PRIVATE KEY", which is what we use: https://github.com/drdoctr/doctr/blob/0f19ff78c8239efcc98d417f36b0a31d9be01ba5/doctr/local.py#L328

peterjc commented 5 years ago

Yes, dotrc configure may use PKCS#8, but there is nothing to stop the user making their own keys - if it is trivial to support I would like to (it looks like just a minor header/footer change at the level we need to worry about).

asmeurer commented 5 years ago

I don't know if we really need to support that, unless it is trivial to do so.

peterjc commented 5 years ago

Took a bit of poking to work this out, but while \ for a space is fine, \n in an environment variable breaks on TravisCI, eventually had to test with a non-secure environment variable in the TravisCI settings to see how it was failing. The key is recorded correctly, but this fails:

Setting environment variables from repository settings
$ export DOC_KEY=-----BEGIN\ OPENSSH\ PRIVATE\ KEY-----\nb3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAACFwAAAAdzc2gtcn\nNhAAAAAwEAAQAAAgEAtlM01FoYLahecnD0Rs7A63A5YVIEmE4WWToML6/DE3QWm+gRBC2x\ncIdcOKd0y0imx3ROVi7GQLvmcADexfj8iuUbfvQwSFRAfJ3Ejs1+Us41FeBi08IEG+eURO\nsRS+mQgjWbjErc/vq5JciQiY1mDOcUOkPD9deQZRdFMdgkoVo2LYwPyBDN8Q0/eOGbby62\na2Tz2KcCW67/41vTUuMciyXStbZBq6xQCCgZhPjP0lR61m7HjslNzlR/Dr2iM9CAGRuEvC\n18h5Nv2jcluri8Q5O+IJlPd+slmt0cEMXzsRIVPjIXz1WvSaNYRdcSZQu8O+jJHMJSC7Oj\n2ZYdEDW3z8I06h+90l/F3HejBI7AE+xXtf1m2c0EHvFPckm+6HVJvCWe0a6BwukcehxgSj\nlGW8XMwux2nG+z/p/MC9wVLAnOEW7GbXBIhxj1HLiezGf/2v4librtzVi5kCpfFYAdhH81\nTZ7u9uUV7xGkCHajKb9BQP6ceSP7MjAVmyV8gXb2LQLwZkLBk0vXawJqNuPHpj0nYZK9eR\nzBW4FJKi2a6K3Cusq1fuvrCUOPctvzBguPVSVzGSVAmDQL78AhZOa7ErKHzgjCWTQYif8A\n79e66m2tV4LRIIFtbFPQxf2rOEl2IfD/datz4T0KZXgYEAsob+R5jbSO7kmFIPHWora07k\nUAAAdAXh0btl4dG7YAAAAHc3NoLXJzYQAAAgEAtlM01FoYLahecnD0Rs7A63A5YVIEmE4W\nWToML6/DE3QWm+gRBC2xcIdcOKd0y0imx3ROVi7GQLvmcADexfj8iuUbfvQwSFRAfJ3Ejs\n1+Us41FeBi08IEG+eUROsRS+mQgjWbjErc/vq5JciQiY1mDOcUOkPD9deQZRdFMdgkoVo2\nLYwPyBDN8Q0/eOGbby62a2Tz2KcCW67/41vTUuMciyXStbZBq6xQCCgZhPjP0lR61m7Hjs\nlNzlR/Dr2iM9CAGRuEvC18h5Nv2jcluri8Q5O+IJlPd+slmt0cEMXzsRIVPjIXz1WvSaNY\nRdcSZQu8O+jJHMJSC7Oj2ZYdEDW3z8I06h+90l/F3HejBI7AE+xXtf1m2c0EHvFPckm+6H\nVJvCWe0a6BwukcehxgSjlGW8XMwux2nG+z/p/MC9wVLAnOEW7GbXBIhxj1HLiezGf/2v4l\nibrtzVi5kCpfFYAdhH81TZ7u9uUV7xGkCHajKb9BQP6ceSP7MjAVmyV8gXb2LQLwZkLBk0\nvXawJqNuPHpj0nYZK9eRzBW4FJKi2a6K3Cusq1fuvrCUOPctvzBguPVSVzGSVAmDQL78Ah\nZOa7ErKHzgjCWTQYif8A79e66m2tV4LRIIFtbFPQxf2rOEl2IfD/datz4T0KZXgYEAsob+\nR5jbSO7kmFIPHWora07kUAAAADAQABAAACAAVjhi80jnxi+o+lWQLyv271/SwsMWiS/kQa\n3BtYtYpQpw5qmH4oH5N03Lvr7x7HAe7pOBuB/GgTZacvz2XIiyZLyTlCXF725oUIYLPbzP\n0DjTwdSq55kIeGm7JBxFwbBiNjFDrcR5XdsTU1rVfzbu24srdaI2OTwHfMbUYg3kxKj0lf\nV2QRYxTZ0GNxP5fUqJ2Z+T8RCJfPH08H+sH7Zm+SaZznSx+Y0zC6hq6QcAwGZS8LTSiiOI\nN9qde0sSOY6QBU+OJa8dBMAkVP1fSj2ySgLEFoZpnjwFPIbzjrsN0t7KTO9DczkB+8DTLb\nPXbxFtoqci9Aaw5fe9tFfYxqGgmkKjxUVwhHGsBHr160UYo27amLArAmqpiyiSsmXEwkdt\n7nQikYbsRh5RA+AHPJ00Pv+83W/Tffx4xjj8WvkiIb7ux/qfXJ2T+F39gdf9M1tAGMwUOW\nd79aTVOnY9Vg4TWzxF3Fs7xUirarYwAlVxd3tj8TUZFfLZeIhY7qs5384NJ/Ndv20w2ss3\nqB585z9dYGPx3J+rv0ePwFf1gX1Yi9amoXcGC36ZgFK96v4B7Vh/VmFAhf7+s/itYowxMH\nOaKmj+2XM8/4GLBm/r4pL7iVzPshysUDr77AOx8mQkZRRaxvsFu4eVk3kSuGfHexxJ7gbt\nmFsiS/1tQ9+ohKYZNNAAABACDYBKEg+eUuHnGFabmmPZj9rf/+0Q+pb1td6rxW8jz16BPo\nLad2wxMIZZsYhEOe/0Hfu4ThNHFCxbIP6m6xFGeyS5NxbGP4M1xePa69/083i/yuy/lPWt\nEgqv5EdmFpfnIoB63kolknP5/bo5jX9QnObXyypEmiibs2yWrzbTzDXtzv7Jy1G/6tWUDW\nS43lQkAweDoQf+ZUJetsk6954Cp8Srps2PCJBki1lxA+yP+PkYzYsZqkg69wBIqDAH1cu9\n1GzNbQdLP8mV+UjhF3fMNjmZMhBqkwmWzHTmNFciSudbWjkjva32n9VqE2lapPNlqrQxt0\n0FvyJiXdzAWp17EAAAEBAOHU/tOWNQhbcZvOqsb9jgXbpC6kKsFK+RrCY5ZiKcMRjaP6tI\nEJHTrCRTpwU899/rBDuW8RtxnboG/6kIkEHvW9EhjG/Z/873TIVOmseB+PZkmMRcdR44iY\n9w5PDe7+p0VQIOEzPEKuofUKdOPdYO6SaHHnix7AXmcqb6+W3SAkKy0wpxXJFmL6EiYE/6\nO8Sxl5RN8tAu/hO/TDBbO8Z4MQzr+2V/5yUAz02gneylzn5bZyFuowRVclDMPGYS5jOV+V\ncrqt4x3rGZR8ojLzZegOWgsi1KINifEmJgTC2/dvbGOOgtSUetzz6nDrp/YgdZuaVK6dRx\nbjwluRvNtyVP8AAAEBAM6uW9MLpYR6wTxa12BgQWDAd7+p0NUsWJI9iWkHCWZbKANqtVnl\ncR/0KJUw4eQonBfQ9QgoYhj+inJoP0bud3X+s6kzL1fjy5Cyy5iA1uwpL7917r3oECI+NV\n0nlc/z1ZBSI1k44h1PODd4/nLB+iNSn8bn2n5tnbcQg5Q5V556VVuVTKHM8LeMK9/sG2UD\nCCFIV4BZGAw9O6LPrebz+KYdb5CNNOYHd/SrRKqWc8P1r2VMi9EHtWVIdmaVdVc/akfmku\n2uAx3jXKIjaDfCqGbAkYUsq3+WisOw6x9xeERITParkyzm14SlVjMvSlB2g1Srr16JlPYn\nxMZGLxYXKLsAAAAIdGVzdCBrZXkBAgM=\n-----END\ OPENSSH\ PRIVATE\ KEY-----

(Note - This was a temp key, created just for testing)

It looks like the slashes are treated as line continuations for the export command, so by the time we come to use it while the spaces can be recovered, the newlines have become just n.

Instead, seems must escape spaces as \ and new lines as \\n when entering them into TravisCI.

asmeurer commented 5 years ago

I tried finishing this over at https://github.com/drdoctr/doctr/pull/350. When I try to encrypt the private key, I get

Traceback (most recent call last):
  File "/Users/aaronmeurer/anaconda3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/aaronmeurer/anaconda3/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/aaronmeurer/Documents/doctr/doctr/__main__.py", line 616, in <module>
    sys.exit(main())
  File "/Users/aaronmeurer/Documents/doctr/doctr/__main__.py", line 613, in main
    return process_args(get_parser(config=config))
  File "/Users/aaronmeurer/Documents/doctr/doctr/__main__.py", line 270, in process_args
    return args.func(args, parser)
  File "/Users/aaronmeurer/Documents/doctr/doctr/__main__.py", line 512, in configure
    travis_token=travis_token, **login_kwargs)
  File "/Users/aaronmeurer/Documents/doctr/doctr/local.py", line 84, in encrypt_variable
    return base64.b64encode(key.encrypt(variable, pad))
  File "/Users/aaronmeurer/anaconda3/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 448, in encrypt
    return _enc_dec_rsa(self._backend, self, plaintext, padding)
  File "/Users/aaronmeurer/anaconda3/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 68, in _enc_dec_rsa
    return _enc_dec_rsa_pkey_ctx(backend, key, data, padding_enum, padding)
  File "/Users/aaronmeurer/anaconda3/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 122, in _enc_dec_rsa_pkey_ctx
    _handle_rsa_enc_dec_error(backend, key)
  File "/Users/aaronmeurer/anaconda3/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 136, in _handle_rsa_enc_dec_error
    "Data too long for key size. Encrypt less data or use a "
ValueError: Data too long for key size. Encrypt less data or use a larger key size.

This is probably the original reason why I didn't include the private key directly in the environment variable.

asmeurer commented 5 years ago

See https://stackoverflow.com/questions/55155642/python-cryptography-data-too-long-for-key-size and https://stackoverflow.com/a/7146463/161801. I did some testing and you can only encrypt 501 bytes with the Travis public key. Any more than that and you get the error.

I think if we want to do this, we need to do the double encryption like we currently do, and put the contents of what would be github_deploy_key.enc in an environment variable. This doesn't really save the user much. Instead of doing a checkout of a file in the repo they would add the contents of said file to their .travis.yml.

peterjc commented 5 years ago

@asmeurer I'm not sure why you are trying to encrypt anything - the goal of what I'm doing is to put the (slash escaped) private key directly into the continuous integration system's moderately secure environment variables settings (something TravisCI offers, and hopefully others too).

This is working for me now using a proof of principle bash script for deployment, see https://github.com/biopython/biopython/pull/2172

asmeurer commented 5 years ago

I didn't try to use the Travis UI. I can't imagine it would work there when it doesn't work in the API, but maybe they do something different there.

The encryption is what makes the key secret. It is encrypted with the public key for the repo. It's the same thing as the travis encrypt command.

asmeurer commented 5 years ago

It does let me paste it in the UI. I haven't tested yet if it actually works. travis encrypt gives an error

data too large - consider using travis encrypt-file or travis env set

peterjc commented 5 years ago

Re: https://docs.travis-ci.com/user/environment-variables/

A common way to customize the build process is to define environment variables, which can be accessed from any stage in your build process.

The best way to define an environment variable depends on what type of information it will contain, and when you need to change it:

if it does not contain sensitive information, might be different for different branches and should be available to forks – add it to your .travis.yml

if it does contain sensitive information, and might be different for different branches – encrypt it and add it to your .travis.yml

if it does contain sensitive information, but is the same for all branches – add it to your Repository Settings

For my use case I don't want special values in .travis.yml as that would block live testing on forked repositories (without a code change in the contents tracked by git, which complicates pull requests etc). Repository Settings is perfect.

If you are willing to trust TravisCI, then you don't need to do anything extra to secure the private key. I'm OK with that, but maybe you're not?

asmeurer commented 5 years ago

I guess we can use https://docs.travis-ci.com/api/#settings-environment-variables. I think this API didn't exist before. I still need to test that it actually works.

peterjc commented 5 years ago

Nice - if the script can set the secret TravisCI variable via the API, that's one less manual step.

Note in the UI, you can't update an existing secret variable - you have to delete and re-add it with the new value.

peterjc commented 4 years ago

FYI, my standalone script version of this has been live and working for over a year now:

https://github.com/biopython/biopython/commit/0bc732cce40879c28b7aea75a6166dfcd7e3989e

Apologies that I have not got back to finishing this pull request for doctr.

drdoctr / doctr

deployment variable in a (secure) environment variable #349