Apostrophe + Docker = Happy Developers and Happy Production

[rwatts3]

Greetings,

I have been working with Docker for quite some time now. I absolutely love Apostrohpe and am amazed with it's power, ease of use and yes power :) .

I believe that implementing Docker with Apostrophe will streamline the productivity of developers in the community that work with Apostrophe because it will create a seamless workflow for production to live. As well as allow a developer to work on their apostrophe project without having to install local software or dependencies to get up and running quickly with Apostrophe.

With that being said currently stagecoach doesn't support docker at the moment, and I believe that maybe extending stagecoach with a set of docker commands would give a great backwards compatible option to those whom are wanting to take advantage of this.

Please let me know if this is something that could be considered.

I'd be happy to dive in and start the process.

-V/R

Ryan

-P.S. not to mention the ability to quickly spin up an apostrophe 0.5 edition , or 0.6 and forward editions with a single command. And the ability to automate the runtime environment as well as run nodemon directly inside the docker container to watch for files on your local machine. Port allocation, separation of core updates and project only updates. The list goes on.

[boutell]

Hi Ryan! Thanks for reaching out. I'm excited about this possibility and I'm not the only one.

Since we're not big Docker users in-house at the moment I'm curious how you see Docker fitting into the picture. Would you use a Docker container to create the server as a persistent critter and then keep deploying to it with Stagecoach to update the actual node app? You could do that with Stagecoach as it stands now, probably. Or would you be using Docker to instantiate a wholly new Node/MongoDB/etc. VM environment on each deployment of the site? That would be significantly different, certainly.

On Tue, Jul 26, 2016 at 1:42 PM, Ryan Watts notifications@github.com wrote:

Greetings,

I have been working with Docker for quite some time now. I absolutely love Apostrohpe and am amazed with it's power, ease of use and yes power :) .

I believe that implementing Docker with Apostrophe will streamline the productivity of developers in the community that work with Apostrophe because it will create a seamless workflow for production to live. As well as allow a developer to work on their apostrophe project without having to install local software or dependencies to get up and running quickly with Apostrophe.

With that being said currently stagecoach doesn't support docker at the moment, and I believe that maybe extending stagecoach with a set of docker commands would give a great backwards compatible option to those whom are wanting to take advantage of this.

Please let me know if this is something that could be considered.

I'd be happy to dive in and start the process.

-V/R

Ryan

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/punkave/apostrophe/issues/619, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB9ffNJOKs2pwO8L4pBGydY9mGoiqnqks5qZkcPgaJpZM4JVZ4x .

THOMAS BOUTELL, SUPPORT LEAD P'UNK AVENUE | (215) 755-1330 | punkave.com

[agilbert]

Hey Ryan, just wanted to jump and echo Tom's excitement around exploring this. At this moment, we're especially keen to explore how Apostrophe can improve support for other platforms and tools that will help streamline developer workflows. It would be great to learn more about some specific use cases that you're seeing on your end.

[rwatts3]

Hi Gentleman,

Thank you for sharing the excitement. I've been working with Docker for at least 2 years now and have watched it evolve over the years. There is an unprecedented list of approaches we could take to use Docker. I'll try to explain a couple different scenarios below. Ideally you can have different approaches depending on the team or group of developers workflow as well as how you want to structure the system is completely up to you but there are definitely some best practices.

If I repeat a few things that are known please forgive me but it's easier to start with the basics in regards to concept before diving into the details.

First there are the different bits that make up docker. Docker Image Think of the image as a base structure for your application. This can be anything from a very minimal ubuntu image, all the way to a centos image with pre-configured settings and software that you know for sure you'll need within your application or any application that will be inheriting this image.

Docker Container The container is just a shell that runs on top of the image and runs your application. You can pass environment settings such as a port, volume mappings to keep data persistent on your local machine or server.

Linking Containers In the instance of Apostrophe we know for a fact that we need imagemagick and we need mongodb. We could easily include image magick within the base or Apostrophe image since we know this is where the main application will live. However we could start a mongodb container with a specified port , and connect our apostrophe-site container to the mongodb container without having to install mongodb directly to the server.

I always stress the importance of using volume mappings for persistent data. Never Ever run a mongodb container in production without mapping the data folder within the container somewhere on the host. If for what ever reason the container fails, or you upgrade the base image of a container and inadvertently blitz the container, with a volume mapping you'll never lose your data.

Docker Compose Think of Docker Compose as the Director of the band. Docker compose can be used to deploy clusters of different containers together to streamline all the bits that are needed for one application. Rather than starting up mongodb, then individually starting our site, as well as any other container we may need "let's say nginx for example". By defining a few settings in docker-compose.yml file at the root of your project. You can run the command docker-compose up and it will build the images needed for each service, and deploy them simultaneously to get the project up and running. Then if you're ready to deploy to production you could run the same command. if you have set your docker-machine's active machine to the production machine. docker-machine env production or what ever you decide to name your production machine.

Now for the fun stuff. For Apostrophe here is what I propose.

• Create a Dockerfile at the root of the apostrophe project. Think of the best base machine typically ubuntu, and write install commands within the Dockerfile for installing things like imagemagick, etc. Expose the port that apostrophe typically will run on should the user mistakenly forget to define a port at least the image will automatically expose the port. In most cases it's ok to do 3000 or 80.
• Now create a docker-compose.yml and or a docker-compose-prod.yml files. Major difference between production and non production files are flags that you would want to run in a production environment for instance changing the volume mapping to a specific location in production where that location may not exist on local running with priveleged =true , which allows the container to have access and make changes on the host as if it were the user directly making those changes. and many others.

I would be happy to start the Dockerfile and yml files to get things running. Where stagecoach would come into play is rather than completely re-writing stagecoach we could give a docker name space so sc-docker some command. The sc-docker would be a wrapper for the docker commands that i've described above. Could detect the dockerfile and yml files in the folder. Personally I really like the way stagecoach orchestrates on the server as it is. Maybe rather than running the individual processes directly on the server we could have docker containers spin up and point to which ever site was deployed for stagecoach and rather than running forever the container would utilize the files from that stagecoach path to run the application.

[rwatts3]

I will setup a repository to kick this off, and let you know what I find. Also it is very beneficial to have tags, or releases/branches when auto generating Dockerfiles. So I could create an image that will check for an environment variable such as apos-version: 0.5 Then use that environment variable to clone the proper tag or branch from apostrophe should the user want to start an application with an older version or migrate their existing workflow to the Docker solution.

[rwatts3]

https://github.com/rwatts3/apostrophe-docker

[rwatts3]

Also I see the main benefit for using Docker in the future is the fact that You guys at Punkave can control dependencies and specifics to the project via the Docker approach. How many issues come up where someone had a specific version of node running, or they didn't want image magick or prefered graphicsmagick or had some port collision they weren't aware of. If you can prebake the ideal environment and allow them to focus on code. This will be best for them.

We could create an apostrophe-docker-sandbox repo, Where there are two folders modules, appfiles and app.js with the docker configuration maybe even db to persist the local mongodb data. Then everything from Apostrophe Core can stay in core within the container. Since we are fetching the files and running the setup, there would be no need for the files to be outside of the box, but if they wanted to they could easily set a mount point to another folder and have the files pull down to their local as well.

[boutell]

I'm really interested to see what you come up with here... we are focused on finishing 2.0.0, which will be npm published in the next few weeks; checkout the unstable branches...

https://github.com/punkave/apostrophe-documentation/tree/unstable https://github.com/punkave/apostrophe/tree/unstable

If you were to get creative with Docker you'd want to base it on those branches.

Also there's the apostrophe-cli module to consider:

https://github.com/punkave/apostrophe-cli

We'd love to pave the road for those who want to use Docker.

On Wed, Jul 27, 2016 at 11:48 AM, Ryan Watts notifications@github.com wrote:

Also I see the main benefit for using Docker in the future is the fact that You guys at Punkave can control dependencies and specifics to the project via the Docker approach. How many issues come up where someone had a specific version of node running, or they didn't want image magick or prefered graphicsmagick or had some port collision they weren't aware of. If you can prebake the ideal environment and allow them to focus on code. This will be best for them.

We could create a apostrophe-docker-sandbox repo, Where there are two folders modules, appfiles and app.js with the docker configuration maybe even db to persist the local mongodb data. Then everything from Apostrophe Core can stay in core within the container. Since we are fetching the files and running the setup, there would be no need for the files to be outside of the box, but if they wanted to they could easily set a mount point to another folder and have the files pull down to their local as well.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/punkave/apostrophe/issues/619#issuecomment-235628925, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB9fVSR0n6mUMKWu00fkNaU20KRgVkQks5qZ32_gaJpZM4JVZ4x .

THOMAS BOUTELL, SUPPORT LEAD P'UNK AVENUE | (215) 755-1330 | punkave.com

[alexgleason]

But isn't Apostrophe more of a dependency than an actual "app"? By its very nature it would require some sort of demo or additional code to even test. I'm wondering what this image will contain.

I suppose there are also images like the Docker PHP image, which is designed to be used as a base image for your PHP projects. But with the nature of Node, it is a trivial task to just use a standard Node base image and include Apostrophe in the project dependencies.

If there were to be an image for Apostrophe, I imagine it would ideally be dead simple.

FROM node
RUN apt-get install imagemagick

The above is just a guess. Give or take a few apt packages.

Better yet would be a Dockerfile included in Apostrophe's scaffolding tools so you can build an image for your own site. Assuming we release an Apostrophe base image, that might look like

FROM apostrophe/apostrophe:latest
COPY . /app

I don't think it makes sense to start up Mongo and all that stuff inside the same image as the app server. The best practice is usually to keep only one process running per container. Spinning up the whole ecosystem could be done with docker-compose, or (my preference) Dokku.

Side note: how do we separate dev from production settings typically in Apostrophe? Do we keep them in separate files or just have a complex app.js? For Docker we probably want to pass the Mongo URI as an environment variable in production.

[boutell]

These are all good questions. Let me ask a naive question of my own: isn't Docker mostly about getting system services in place, setting the stage as it were? Is it worth blowing away the container on every deploy of the site, or should Docker just be used to create the permanent environment and stagecoach used for ordinary deploys after that, unless a move to another environment is desired?

As for separate containers, you can separate mongodb from node, as long as you don't introduce a lot of latency between them — that would really kill performance with Apostrophe. The file system is another story... by default Apostrophe expects to be able to store media files on disk in a durable manner, I'm not sure how that impacts things (or doesn't) with Docker. However you can configure Apostrophe to use S3 or something compatible instead.

On Wed, Sep 7, 2016 at 2:18 AM, Alex Gleason notifications@github.com wrote:

But isn't Apostrophe more of a dependency than an actual "app"? By its very nature it would require some sort of demo or additional code to even test. I'm wondering what this image will contain.

I suppose there are also images like the Docker PHP image, which is designed to be used as a base image for your PHP projects. But with the nature of Node, it is a trivial task to just use a standard Node base image and include Apostrophe in the project dependencies.

If there were to be an image for Apostrophe, I imagine it would ideally be dead simple.

FROM node RUN apt-get install image magic

The above is just a guess. Give or take a few apt packages.

Better yet would be a Dockerfile included in Apostrophe's scaffolding tools so you can build an image for your own site. Assuming we release an Apostrophe base image, that might look like

FROM apostrophe/apostrophe:latest COPY . /app

I don't think it makes sense to start up Mongo and all that stuff inside the same image as the app server. The best practice is usually to keep only one process running per container. Spinning up the whole ecosystem could be done with docker-compose, or (my preference) Dokku.

Side note: how do we separate dev from production settings typically in Apostrophe? Do we keep them in separate files or just have a complex app.js? For Docker we probably want to pass the Mongo URI as an environment variable in production.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/punkave/apostrophe/issues/619#issuecomment-245185865, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB9fRYUxlFIQYoWUhhni5wzmLSWPfZMks5qnldSgaJpZM4JVZ4x .

THOMAS BOUTELL, SUPPORT LEAD P'UNK AVENUE | (215) 755-1330 | punkave.com

[alexgleason]

Typically Docker containers contain "apps", or they define a base image for an app to inherit from.

myclientsite.com might be one Docker image. It's done this way so you can rest assured that the site will function without any extra configuration, and it's sandboxed so you can destroy it without harming other software on the server. It is worth every site having its own image because images are constructed out of the equivalent of git commits, so if you base two images on the same image the system will compress them.

Btw I live like three blocks from Punk Ave, would be cool to meet up if you have time. I'm coming from Wagtail and pretty excited about Apostrophe.

[rwatts3]

The image would more so be treated as the "Ideal " environment that apostrophe needs to run smoothly for all operating systems.

I would see the image containing a directory of "apps" , with npm, node, imagemagick, and forever already installed with the recommended versions. There could even be a production image , or services image that contains nginx, and mechanic. Then we would use docker-compose to point our local directory to the apps directory inside the container generated from the image. Once you spin up the containers with docker compose you get a fully working environment for windows, linux, and mac.

I would suggest having two different docker compose files one for production and one for local. The production environment would use forever.

A docker namespace in the stagecoach module could be used as a wrapper for docker compose commands. So if you wanted to deploy mongodb, nginx/mechanic, and your application. You would still copy your app directory to the native server , then point your docker container to that directory and use this as the base of your application with all of the prerequisites installed within the container/image layer.

[alexgleason]

How can we know the "ideal environment"?

[rwatts3]

I Agree with @boutell to help reduce the latency , create a link between your node container, and the mongo container and have them talk directly to each other.

[rwatts3]

I usually determine the ideal environment based on the instructions of what the application or service is suggesting.

So if Punkave is recommending a specific node version, with a specific version of npm, and a specific version of imagemagick. You would set your image up to contain that specific environment. If they say Apostrophe runs best on ubuntu 14 vs. centos , then you would use either one of those operating systems as your base. It really depends on the installation instructions of Apostrophe.

[rwatts3]

@alexgleason I like the idea of using a Dockerfile for continioius deployment in a production environment. It totally makes since. In the Meteor community a lot of devs use an image that contains something like a base tag, an onbuild tag, as well as a production and or dev tag. If you were to deploy using the onbuild tag. It could contain instructions to use the base image , and build the app by copying the local source directly to the container, and using that rather than pointing to code on the production server.

The base container could be used to hold all of apostrophe's code.

One of the benefits I see using docker is the fact, that you can keep common modules that you share across multiple clients in one docker instance. And share them across different sites. These could be private modules that you wish to not share publicly with the community.

[rwatts3]

@boutell You could define a local mount , for the file storage. I would even recommend having the entire code stored on the server just as it already does with stagecoach, for persistence. The only thing docker would do, is look for the code's directory and mount it inside the container. And house all of the prerequisites for apostrophe. I would say we can start with taking the manual steps in production and building the image/container based on that. Then just leave the source as you are already doing it.

[boutell]

That makes sense.

So would we still use sc-deploy as we do now for deployment of each new update of the site, to an existing container? Or would the entire business rebuild (with some optimizations, I gather, that make that fast...) on each deploy? What happens to durable files like public/uploads?

On Wed, Sep 7, 2016 at 11:11 AM, Ryan Watts notifications@github.com wrote:

I usually determine the ideal environment based on the instructions of what the application or service is suggesting.

So if Punkave is recommending a specific node version, with a specific version of npm, and a specific version of imagemagick. You would set your image up to contain that specific environment. If they say Apostrophe runs best on ubuntu 14 vs. centos , then you would use either one of those operating systems as your base. It really depends on the installation instructions of Apostrophe.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/punkave/apostrophe/issues/619#issuecomment-245312447, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB9fdn3UaQFOjYtPYFx8qrqe8p4GP03ks5qntQegaJpZM4JVZ4x .

THOMAS BOUTELL, SUPPORT LEAD P'UNK AVENUE | (215) 755-1330 | punkave.com

[boutell]

Would love to see someone take a rough and ready pass at this so we can experiment with that!

On Wed, Sep 7, 2016 at 11:19 AM, Ryan Watts notifications@github.com wrote:

@boutell https://github.com/boutell You could define a local mount , for the file storage. I would even recommend having the entire code stored on the server just as it already does with stagecoach, for persistence. The only thing docker would do, is look for the code in the upload path. And house all of the prerequisites for apostrophe. I would say we can start with taking the manual steps in production and building the image/container based on that. Then just leave the source as you are already doing it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/punkave/apostrophe/issues/619#issuecomment-245315208, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB9fcNLoEGQhkR6Q7dddwl_KvbXPM52ks5qntYdgaJpZM4JVZ4x .

THOMAS BOUTELL, SUPPORT LEAD P'UNK AVENUE | (215) 755-1330 | punkave.com

[rwatts3]

Truthfully I am a huge fan of sc-deploy and I think it does it's job just fine. May be we can introduce a docker flag --docker or a docker name to accomplish this task.

I also submitted an issue to the stagecoach repo about possibly submitting a PR to allow stagecoach to work on all platforms.

I've written a few cli tools using node and shelljs. Maybe this could be a good option to introduce the docker commands, while maintaining the existing sc-deploy logic.

[rwatts3]

Durable files like public/uploads could be stored in a volume on the server, and included within the container by using a volume argument. This ensures when the container is torn down and rebuilt, so long as you point to that same directory the files will be persistent.

Some in the docker community recommend using data containers to store things like that. I advise against it because a data container an be destroyed the same way any other container is destroyed. For things like this or persistent mongodb data it's always a good idea to map it directly to the server.

[rwatts3]

Wanted to share this,

A first crack at making stagecoach able to run on any operating system, which will be the foundation for the docker options and deployment.

This is running on windows. Still a good deal to go.

[rwatts3]

https://github.com/rwatts3/stagecoach/tree/stagecoach2

[alexgleason]

Here's a Dockerfile that will work with any Apostrophe project.

FROM node:6.5.0

# Create app directory
RUN mkdir -p /app
WORKDIR /app

# Bundle app source
COPY . /app
RUN npm install

EXPOSE 3000
CMD [ "npm", "start" ]

Just include it as Dockerfile in the base of your project and run docker build -t punkave/test-project . to build the image. Make sure npm run start runs the server, because this relies on it. I also recommend including a .dockerignore file to prevent unnecessary files from being copied into the image.

At this point it's really up to you to configure it and bring it up depending on how you set up MongoDB. But the following steps can get you started.

Start by including environment variables to connect to Mongo in your app.js.

    'apostrophe-db': {
      uri: 'mongodb://' + process.env.MONGODB_PORT_27017_TCP_ADDR + ':' + process.env.MONGODB_PORT_27017_TCP_PORT + '/mydb'
    },

Then rebuild the image, pull the Mongo image, run the Mongo image, and then run the the Apostrophe image you just built, linking it to the Mongo image.

$ docker build -t punkave/test-project .
$ docker pull mongo
$ docker run -d --name apos-test-db mongo
$ docker run --link=apos-test-db:mongodb -p 3000:3000 punkave/test-project

If you run into problems or these instructions aren't clear enough, see this guide to using Docker with MongoDB and Node.

The Dockerfile is really simple. The Node app it bases its image from even has ImageMagick already so that's not a concern.

A really opinionated official Apostrophe Dockerfile could look like this:

FROM node:6.5.0

EXPOSE 3000
CMD [ "npm", "start" ]

While the project Dockerfiles look like this:

FROM punkave/apostrophe:latest

# Create app directory
RUN mkdir -p /app
WORKDIR /app

# Bundle app source
COPY . /app
RUN npm install

Again, having an Apostrophe Dockerfile seems kind of pointless when it needs literally no extra dependencies? It might be good just as a placeholder alone, and with a good readme attached. But I'm not sure it's worth obfuscating the Node image just to call it something different.

[alexgleason]

Well, Docker supports uploading images with arbitrary tags. Tags represent variations in the image, like different versions or different configurations. If different versions of Apostrophe require different Node versions or dependencies then it might be worth it.

For instance, perhaps a user can do this:

FROM punkave/apostrophe:0.5

or this:

FROM punkave/apostrophe:2.0.0

So, in that sense maybe it could be worth having dead-simple Dockerfiles as proxies.

[boutell]

Closing this because we have a Docker tutorial:

http://apostrophecms.org/docs/tutorials/howtos/docker.html

Feel free to open new issues for improvements, additions, etc. Didn't want this to get too unwieldy.