search

mattkrick / meatier

:hamburger: like meteor, but meatier :hamburger:
3.05k stars 173 forks source link

How to deploy in production? -- Options and best pratices #67

Closed 0o-de-lally closed 8 years ago

0o-de-lally commented 8 years ago

Really enjoying meatier. The production bundling makes a lot of sense. Though I'd like a tool like mupx (using docker) to deploy a production app. Any thought on hacking mupx for this?

bartekus commented 8 years ago

We are currently working with azk.io in the area of common development environment and deployment. The whole process is nearly completed but few rough spots need to be ironed out before the release. However I'll look into mupx as well since it would be a shame to omit any potential great tool that could also make meatier a better build all together.

0o-de-lally commented 8 years ago

Never heard of Azk.io, seems like it could work well. Good job I really like the meatier project...

patrickleet commented 8 years ago

@keyscores check out my PR here: https://github.com/mattkrick/meatier/pull/64

These were the changes that were required to deploy meatier on modulus.io using codeship as a CI/build server, and a Compose RethinkDB deployment.

Using this process, deploying is as simple as git push origin master.

All you need to do is go to codeship.com, select your repository with the github integration, and select modulus for the deployment integration. You can use their command line tool to generate an API token. (haven't tried others but nodejitsu doesn't seem to be accepting new clients, and heroku runs the build on their side typically, and that isn't necessary with codeship handling it)

For the test script you can use this:

# By default we use the Node.js version set in your package.json or the latest
# version from the 0.10 release
#
# You can use nvm to install any Node.js (or io.js) version you require.
# nvm install 4.0
nvm install 5.4.1
npm install
# we want to build the db for testing
NODE_ENV=testing npm run build:db
# build client and server with production settings (handled by npm tasks)
npm run build:client
npm run build:server

I actually started with the azk.io branch @Bartekus made when going down this route, but azk.io seemed to be pretty lacking in the deployment capabilities. For local development it is great, but I couldn't figure out how to just deploy one "system", and not the db for example. I'm looking forward to see what you can make of it @Bartekus.

paralin commented 8 years ago

I've just discovered meatier and azk. As a long time Meteor user and with a Docker + Kubernetes background I'm interested in writing easy ways to deploy stuff to Kubernetes.

I think it's too specific to build this kind of functionality directly into Meatier. But building it into Azk might make sense. What do you guys think?

bartekus commented 8 years ago

I'm sorry for the delay @patrickleet , I got it almost done but I got just one more thing to figure out before it will be polished enough for everybody's use... Personally I'm all for dockerizing meatier, in fact we have started a side project as r3stack to eventually do just that but it's definitely the right direction @paralin so I'm more than happy to assist you and infact do encourage you to pursue this idea!

patrickleet commented 8 years ago

No worries about the delay, just wanted to help anyone who is trying to get things up and running ASAP before the next build is ready :)

bartekus commented 8 years ago

alright, here is what we are working on, r3stack it still need the changes that you made applied to it, but it has the most completed deployment structure and all updates as well as we're trying to polish all aspects of it into it's own framework like architecture. Tho it's not really a framework, more like a crazy experiment to get kids onboard of developing cool things :)

bartekus commented 8 years ago

fyi, I'm fairly sure webpack is cause of most of my problems at the moment, I downgraded from beta 2.0.7 to 2.0.6 to 2.0.5 to 2.0.4... so on back to 2.0.2 lol it doesn't like
`compiler: { hash_type: 'hash', stats: { chunks: false, chunkModules: false,

colors: true,

},

},`

lol, for now we'll use 2.0.2 and I'll talk to Kosta to find out why all of sudden colors we're taken out of the stats compiler. :/

patrickleet commented 8 years ago

Interesting.. why go to a new repository for what looks to be pretty much the same thing? I mean, it's really hard to tell as there is no diff.. Is this going to be merged back into Meatier?

FWIW, the "meatier" name is clever, and communicates the goals pretty clearly. Also, I think the cheekiness draws people to it, especially as frustration with Meteor rises. This repo has gained 500 or so stars in the past week or so since I've found it, seems like if you want the most people using it as possible you'd stick with the repo that's already doing well?

bartekus commented 8 years ago

That is true, but our alternative goal is not to live in meteors shadow but rather create alternative stack for hackers to train hackers... Meteor was commercial product, r3stack is by hackers for hackers, our gift to the anonymous movement if you will. And in that respect, the stars mean very little as it's the idea that it represents. Beside Meatier is perfect as it is and aside from gradual updates and changes it won't drastically morph into something else as it is in store for r3stack. Also by being associated with meteor we could be forced to follow a pattern that we might not like to follow a year from now on.

bartekus commented 8 years ago

@patrickleet But your notion is duly noted... So I'll implement your changes immediately and integrate what I got so far working with r3stack into meatier so that a broader range of people can benefit from these improvements. Thank you for your objectivity and understanding.

bartekus commented 8 years ago

As promissed your changes have been implemented and tested ok on the Araphel branch

bartekus commented 8 years ago

also Meatier is now online

paralin commented 8 years ago

I'll start working towards some sort of deployment model targeting Kubernetes via Azk with Meatier as an example. Mostly because I want something like this for my own work.

bartekus commented 8 years ago

Thats very cool, I honestly didn't know about Kubernetes so if you need any help, do let me know, I'm certainly interested in learning more about it!

mattkrick commented 8 years ago

@Bartekus the webpack problems are on me, the API is slightly in flux for v2 and I need to update the configs. See https://github.com/webpack/webpack/issues/1947 as Tobias explains exactly how to fix it

bartekus commented 8 years ago

Ah that explains it, I was puzzled while looking at the docs but now it's much clearer. If you need any assistance do let me know, I'll be working on other dependencies as I've got update branch going also.

bartekus commented 8 years ago

Let's keep this focused. The issue [How to deploy in production? #67] issue is resolved as of #95 I recommend closing this ticket and reopening one ticket per each outstanding issue.

wenzowski commented 8 years ago

For the git push crowd there's also dokku and its rethinkdb plugin. All that takes is some process.env.PORT swapping and NPM_CONFIG_PRODUCTION=false for cedarish to install webpack.

ncrmro commented 8 years ago

This isn't my exact config i'm using nor would I use it in prod without due diligence. I use tutum with a load balancer as it's much easier to configure and have image's automatically redeploy when triggered by a push to the docker hub image repo either from a manual push or from continuous integration (circle ci builds and pushs the docker image after a push to git, not much testing atm.)

Still need either look more into database volume backup procedure or just use compose (the database service).

Deploying with the Docker Toolbox

eg

Create a local docker environment in virtual box and a remote. Edit if virtualbox settings if need be.

 docker-machine create -d virtualbox \
     --virtualbox-cpu-count "3" \
     --virtualbox-memory "4096" \
     default

docker-machine env default 

  export DOCKER_TLS_VERIFY="1"
  export DOCKER_HOST="tcp://192.168.99.100:2376"
  export DOCKER_CERT_PATH="/Users/ncrmro/.docker/machine/machines/default"
  export DOCKER_MACHINE_NAME="default"
  # Run this command to configure your shell: 
  # eval $(docker-machine env default)

Build the docker image using this dockerfile

FROM node:5.6

# use changes to package.json to force Docker not to use the cache
# when we change our application's nodejs dependencies:
ADD package.json /tmp/package.json
RUN cd /tmp && npm install
RUN mkdir -p /opt/app && cp -a /tmp/node_modules /opt/app/

# From here we load our application's code in, therefore the previous docker
# "layer" thats been cached will be used if possible
WORKDIR /opt/app
ADD . /opt/app

RUN npm run build

ENV DATABASE_HOST=localhost \
    DATABASE_PORT=28015 \
    GRAPHQL_HOST=localhost \
    GRAPHQL_PROTOCOL=http

EXPOSE 3000

ENTRYPOINT ["npm"]

CMD ["run", "prod"]

docker build -t ncrmro/meatier:master .

Push the image docker login --username=yourhubusername --email=youremail@company.com

docker push ncrmro/meatier:develop

Create a new prod docker machine.

docker-machine create --driver digitalocean \
     --digitalocean-access-token=abc123 \
     new-docker-machine-name

Change the docker environment to the new machine.

docker-machine env new-docker-machine-name

Docker-compose file consists of a base environment for development (Webpack Not refreshing correctly, something something rsync)

The database url

This will pull the rethinkdb and build your app. We want to over ride this in production eg.

docker-compose.yml

version: '2'

services:
  db:
    image: rethinkdb
    ports:
        - "8080:8080"
        - "28015"
        - "29015"
    volumes:
        - /data

  app:
    build: .
    volumes:
      - ./src/:/opt/app/src
    ports:
        - "80:3000"
    command: start
    environment:
      DATABASE_HOST: kaasol_db_1
      GRAPHQL_HOST: kaasol_app_1
      GRAPHQL_PROTOCOL: http

docker-compose.prod.yml

version: '2'

services:
  app:
    image: ncrmro/meatier:develop

docker-compose -f docker-compose.yml -f docker-compose.prod.yml up

mattkrick commented 8 years ago

Rather than create a new issue, let's just get all the options out there. azk.io seemed like the easiest, most straightforward, but I'd love to see all the options we have & compare ease of use & power of each.

ncrmro commented 8 years ago

Some one one gitter asked about how tutum is working. I miss the heroku push and forget

Here is a circle.yml file that will build and run a docker image from a github repo. (no tests atm just makes sure everything starts up etc and builds the image)

Continuous Integration with CircleCI and Docker.

Circle.yml

Updated with better environment variable passthrough.

machine:
  environment:
  services:
    - docker

dependencies:
  override:
    - docker info
    - docker build -t $CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME:$CIRCLE_BRANCH .

test:
  override:
    - docker run -d $CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME:$CIRCLE_BRANCH

deployment:
  hub:
    branch: master
    commands:
      - docker login -e $DOCKER_EMAIL -u $DOCKER_USER -p $DOCKER_PASSWORD
      - docker push $CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME:$CIRCLE_BRANCH

deployment:
  hub:
    branch: develop
    commands:
      - docker login -e $DOCKER_EMAIL -u $DOCKER_USER -p $DOCKER_PASSWORD
      - docker push $CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME:$CIRCLE_BRANCH

Issues

Contentious Deployment

Watchtower

Tutum

If you use a tutum, add the docker hub repo on their site. Set up the node-cluster (I used one machine atm 1gb ram single core on aws (aws free tier works great) or digital ocean.

tutum-node

The node cluster should be tagged the project name and the development branch I have develop and master. You can see this in the tutum-stackfile below.

Tutum stack-file below.

Once you set this up you'll need a redeploy trigger for the web container. Then you add this trigger url to your dockerhub so once new images are pushed it will trigger tutum to pull the latest image

web-redeploytrigger copy

Tutum-stackfile.yml

lb:
  image: 'tutum/haproxy:latest'
  links:
    - web
  ports:
    - '80:80'
  roles:
    - global
  tags:
    - develop
    - meatier
rethinkdb:
  image: 'rethinkdb:latest'
  ports:
    - '8080:8080'
  tags:
    - develop
    - meatier
web:
  image: 'ncrmro/meatier:develop'
  environment:
    - DATABASE_HOST=rethinkdb
    - GRAPHQL_HOST=dev.meatier.com
  links:
    - rethinkdb
  tags:
    - develop
    - meatier

Finally if using tutum they will provide a dynamic dns hostname so if you make a new node cluster you won't have to switch out dns settings.

Make your domain name cname point to lb.meatier-develop.ncrmro.svc.tutum.io

A pretty stock version of the project can be found hosted at http://dev.jtronics.exchange/ The circle ci build log for the most recent build. https://circleci.com/gh/ncrmro/jtx/74

wenzowski commented 8 years ago

If you want I can PR dokku instructions. It builds with cedarish so it is git push Heroku-easy.

mattkrick commented 8 years ago

@wenzowski sounds good! Instead of a PR could you paste a basic how-to/issues like @ncrmro?