search

databio / bioterms

Content and Dockerfile for the bioterms.org site
http://bioterms.org
1 stars 0 forks source link

bioterms deploy strategy #3

Closed nsheff closed 5 years ago

nsheff commented 5 years ago

after discussion:

it should be reasonably easy to develop locally, and then to deploy updates.

nsheff commented 5 years ago

container instructions worked perfectly for me, thanks!

nsheff commented 5 years ago

So, I am able to run things locally -- but there is no information on how to deploy this on bioterms.org.

oddodaoddo commented 5 years ago

@nmagee : Can we set up a pipeline that deploys bioterms.org on a push to the git repo? Thanks!

nmagee commented 5 years ago

@nsheff I'm ready to push this but could you please help me w/ two setup steps:

  1. Add me as a contributor to this repo? (It won't let me push for some reason)
  2. Hop into https://travis-ci.org/organizations/databio/repositories and enable this repo in Travis?

Container builds will happen automatically from there and Travis will send an SQS message. The cluster will then poll for messages and deploy any new container versions into the cluster.

nsheff commented 5 years ago

done!

nmagee commented 5 years ago

Thanks! Rolling this out now ...

nmagee commented 5 years ago

Ah, sorry @nsheff if you're able to give me repo-level permissions within Travis too, that would help. I need to access the settings to set some environment variables for the builds.

nsheff commented 5 years ago

I actually do not know how to do that and can't find any information on permissions of travis repos...

nmagee commented 5 years ago

I was just looking for it too and haven’t a clue ;-) I think they are actually inherited over from GitHub.

But for whatever reason, travis just gave me permissions. Maybe it takes a bit of time to propagate over?

[cid:image001.png@01D48E1A.27B675B0]

From: Nathan Sheffield notifications@github.com Reply-To: databio/bioterms reply@reply.github.com Date: Friday, December 7, 2018 at 10:42 AM To: databio/bioterms bioterms@noreply.github.com Cc: Neal Magee nem2p@virginia.edu, Assign assign@noreply.github.com Subject: Re: [databio/bioterms] bioterms deploy strategy (#3)

I actually do not know how to do that and can't find any information on permissions of travis repos...

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/databio/bioterms/issues/3#issuecomment-445271925, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAqtluD0jq1n8JNYkEH4BGT_dqkf_V3Qks5u2oyBgaJpZM4YW7A3.

nmagee commented 5 years ago

Okay this is all set. Look at .travis.yml to see under the hood, but basically Travis logs into Docker Hub, builds the container, pushes it up, and then sends an SQS message in AWS.

From there, a polling script runs in cron on the development node. When it sees an SQS message it updates the container. (In DC/OS this will be more seamless, to iteratively fold the new image into a running service. But in this case I'm just killing the bioterms container and relaunching it, with a downtime of probably around 0.5s.)

#!/bin/bash

set -e

QURL="https://sqs.us-east-1.amazonaws.com/474683445819/bioterms-updates"

QCOUNT=`/bin/aws sqs get-queue-attributes --queue-url "$QURL" --attribute-names "ApproximateNumberOfMessages" | /bin/jq -r .Attributes.ApproximateNumberOfMessages`
echo $QCOUNT

if [ "$QCOUNT" -gt 0 ]; then

  # get msg from the queue
  RAW=`/bin/aws sqs receive-message \
    --max-number-of-messages 1 \
    --queue-url "$QURL" \
    --wait-time-seconds 20`;

  # new container image.
  /bin/docker pull databio/bioterms:latest;
  /bin/docker stop bioterms;
  /bin/docker run -d --rm --name bioterms -p 8889:80 databio/bioterms:latest;

  # delete the message
  RECEIPTHANDLE=`echo $RAW | /bin/jq -r .Messages[0].ReceiptHandle`;
  /bin/aws sqs delete-message \
    --queue-url "$QURL" \
    --receipt-handle "$RECEIPTHANDLE";

else

  # Do nothing.
  echo "No messages"
  exit 0

fi

exit 0
nsheff commented 5 years ago

in my test, a push to the github repository did not trigger an automatic travis build.

nmagee commented 5 years ago

@nsheff this should be running smoothly for you. I checked and your build from yesterday had gone through in Travis but the cron job ran only every 5 mins, so there was a bit of a delay.

Give it a whirl and see if you are getting ok deployments now?

oddodaoddo commented 5 years ago

Depending on how long the build takes and other possible delays on the system, you got to be careful with the timing on the cron job ;). Other than that, looks good :)

nsheff commented 5 years ago

I see. I didn't realize it was a cron and thought it was using a git hook.

nsheff commented 5 years ago

Question for the experts: Right now for development I'm using this code to re-start my container after I make a change:

First I re-build the container with docker build, then I do this:

docker stop bioterms; docker rm bioterms; docker run -dit --name bioterms -p 8889:80 bioterms-docker-image:latest

Is that the most streamlined way to do it?

nmagee commented 5 years ago

That’s a perfectly fine wait to do it, yes. If you prefer, you can change that 8889 to whatever local port you want to preview the site under.

Neal

On Dec 17, 2018, at 9:44 AM, Nathan Sheffield notifications@github.com<mailto:notifications@github.com> wrote:

Question for the experts: Right now for development I'm using this code to re-start my container after I make a change:

First I re-build the container with docker build, then I do this:

docker stop bioterms; docker rm bioterms; docker run -dit --name bioterms -p 8889:80 bioterms-docker-image:latest

Is that the most streamlined way to do it?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHubhttps://github.com/databio/bioterms/issues/3#issuecomment-447869764, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAqtlpMtfxfgOEfLhUs_vejQInyD0RJFks5u563igaJpZM4YW7A3.

nmagee commented 5 years ago

Nathan, I was on my phone when I first replied and didn’t notice one little detail: use the -d flag (not -dit) when running it in “detached” mode. The “-it” flag is for interactive/pseudo-tty mode when you want to run and enter the shell of a container.

You can also be lazy (like me) and add a –rm flag on your run command, so you don’t need to run the “docker rm bioterms” command every time.

docker build -t databio/bioterms:latest . ; docker stop bioterms; docker run -d --rm --name bioterms -p 8889:80 databio/bioterms:latest;

Or tag however useful, :dev or :dev238. Your git push will initiate a build and release to databio/bioterms:latest.

Neal

From: Nathan Sheffield notifications@github.com Reply-To: databio/bioterms reply@reply.github.com Date: Monday, December 17, 2018 at 9:44 AM To: databio/bioterms bioterms@noreply.github.com Cc: Neal Magee nem2p@virginia.edu, State change state_change@noreply.github.com Subject: Re: [databio/bioterms] bioterms deploy strategy (#3)

Question for the experts: Right now for development I'm using this code to re-start my container after I make a change:

First I re-build the container with docker build, then I do this:

docker stop bioterms; docker rm bioterms; docker run -dit --name bioterms -p 8889:80 bioterms-docker-image:latest

Is that the most streamlined way to do it?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHubhttps://github.com/databio/bioterms/issues/3#issuecomment-447869764, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAqtlpMtfxfgOEfLhUs_vejQInyD0RJFks5u563igaJpZM4YW7A3.