Errors in docker entrypoint script

[brodkin]

I'm consistently seeing the following errors when starting the container:

Jun 18 02:58:09 coreos-thin-1 docker[17200]: awk: cannot open /tmp/settings.php (No such file or directory)
Jun 18 02:58:09 coreos-thin-1 docker[17200]: awk: cannot open /tmp/settings.php (No such file or directory)
Jun 18 02:58:09 coreos-thin-1 docker[17200]: awk: cannot open /tmp/settings.php (No such file or directory)
Jun 18 02:58:09 coreos-thin-1 docker[17200]: find: missing argument to `-exec'
Jun 18 02:58:09 coreos-thin-1 docker[17200]: find: unknown predicate `--max-depth=1'

The issue here may be that I simply do not understand what the intended function of the script, but it occurs to me that the order of events is currently incorrect.

The script currently looks like this on an existing install:

    # Run a database backup
    dbuser=$(awk -F "'" '/\$dbuser/ {print $2}' /tmp/settings.php)
    dbpass=$(awk -F "'" '/\$dbpass/ {print $2}' /tmp/settings.php)
    dbname=$(awk -F "'" '/\$dbpass/ {print $2}' /tmp/settings.php)

    mysqldump -h mysql -u $dbuser -p$dbpass $dbname | gzip > /var/www/backup/$(date +"%Y-%m-%d_%H-%M-%S").sql.gz

    # Backup the settings.php file
    mv /var/www/xibo/web/settings.php /tmp/settings.php

    # Delete the old install EXCEPT the library directory
    find /var/www/xibo ! -name library -type d -exec rm -rf {};
    find /var/www/xibo -type f --max-depth=1 -exec rm -f {};

    # Replace settings
    mv /tmp/settings.php /var/www/xibo/web/settings.php

We're currently trying to awk /tmp/settings.php even though we do not create that file until after the DB dump.

A little further down both finds have syntax errors. The first one can be explained by a bad termination and the second is an extra hyphen since it should just be -maxdepth.

[alexharrington]

OK I've made those changes.

What I don't understand is why you get that at every start of the container. That whole section is within a block which only gets run if the /CMS-FLAG file still exists (which indicates to us that the container and hence Xibo version was upgraded and the whole bootstrapping process needs to be re-run)?

What should happen, is once the CMS is installed/upgraded, that file is removed, and then the bootstrapping process is skipped on subsequent starts of the container. You don't want to be destroying the container and creating it fresh at each boot. You stop it when you're done and start it when you want it back.

Thanks for pointing those out. I guess it may be a difference in the version of find on CoreOS as I don't see those errors on our test server - or on my dev box.

[brodkin]

It's interesting that you aren't seeing the find errors as find is supplied by the container, not the host. Is it possible that the launcher is masking that output? FWIW, the error is not fatal--just something I noticed.

As for the upgrade issue, I'm still working through it. The pattern right now is that I can run everything without issue, restart, exit with code 127, restart, back to normal. I think what you mentioned about the CMS-FLAG file is a clue...

When we build docker containers for distributed operating systems we assume that as long as data volumes are persisted, the container can be destroyed without consequence. That allows for a node to fail and another to pick up its work. How dependent is Xibo on data stored outside of the mounted volumes? In the case of the CMS-FLAG, for instance, it would be preferable if the file was stored at /var/www/xibo/CMS-FLAG instead since this would survive on the host OS.

[alexharrington]

CMS-FLAG is just there to notify the entrypoint script that the container was rebuilt and/or the Xibo version was upgraded, and therefore it needs to perform some setup (including taking a backup of the database if it already exists, before the CMS makes any changes during an upgrade routine). It won't cause you a problem running that at every container start, but it will cost you a fair chunk of time - especially as that database grows in size. Using our launcher system - which is how the container is really designed to be used - won't encounter that, because the container is only ever intended to be destroyed at Xibo version upgrade time. Day to day, the container is just started and stopped as required.

Moving the CMS-FLAG to a volume defeats its purpose as it doesn't allow us to know if it's the first boot of a fresh container.

Everything Xibo needs to run (database, CMS files, backups etc) are stored on mounted volumes, so you can run:

launcher bootstrap Do stuff in the CMS launcher destroy launcher bootstrap

and that will spin up the containers, create the library, CMS files, database etc. Then destroy those containers completely and remove them, and then recreate them with the data from the original containers intact.

How you apply that logic to your own Docker hosting system will be an exercise for you to implement. You can see the Docker commands we're running directly though so I'd imagine it to be fairly straight forward.

What's the output you're seeing when you get the exit code 127?

[brodkin]

I guess that's what I'm suggesting... I don't want to be too preachy since I truly believe that developers are best equipped to make the right decision for their project, but being that Xibo is committing to continued use of Docker into the future, I'd feel remiss to not at least comment as someone who actively maintains about 20 docker containers. Here's my two cents, take it or leave it:

Docker containers are fundamentally disposable so I would encourage you guys to not maintain any sort of state in the container. As things currently stand containers are built, run, upgraded, and run again. This works well enough with the launcher, as you have suggested, but it goes against the way that most people expect Docker containers to behave. Here are the changes that I would make:

• The entire CMS should come preloaded on the container and it should never be modified in any way--never even by the entrypoint script. If a user wants to upgrade, they pull the new version, destroy the old container and run the new one. Modifying a container to upgrade from one version to another is treating the container like it's a VM, which it is not.
• All environment variables are accessed directly within PHP via $SERVER, eliminating the need to write the configuration into a file.
• Stop storing the mysql root password in plain text--it's just bad security.

What I've laid out here is the design pattern used by almost every major Docker container. You can see it at work in each and every one of the official docker images on Docker Hub: PHP, mySQL, Redis, Wordpress, Nginx, etc.

If you want to switch to a new version of Wordpress, for instance, you just pull a new version and run it over your existing data, and it just runs! You can look at their entrypoint; they only write a couple environment vars to file and handle first run, and that's it.

Alright, off my soapbox, but give it some thought. You guys have done some amazing work on Xibo and my goal is not to diminish that, but rather to help you guys take the next step toward creating some really solid containers.

[alexharrington]

I'm not sure you're understanding what that code is doing fully. I'm not storing any state beyond whether or not it's the first time the container has been started. If you replace the container, there's no issue with that. The main reason to know that is that if the Xibo version has been upgraded (by pulling the container for the newer version and applying it to the existing volumes as I think you're suggesting Wordpress does), then there are things that have to happen external to Xibo to your data to support the upgrade process. If I can't figure out if the container is replaced, by one single state file, then we have to introduce an additional step by which the container is started in Upgrade mode to do those things. Unfortunately, we can't rely on the end user to take backups before upgrades etc.

Re storing the password in a flat file, we do that for the password used to access the database so what's the difference? I don't see how it's any less secure than having it in an environment variable which is what the norm seems to be for Docker? I appreciate it's not ideal, but again not storing it there means storing it in the launcher or somewhere else and passing it in?

I'm very happy to accept suggestions, and we did look extensively at Wordpress etc.

The entrypoint script simply installs the CMS to the persistent volume in a sane way. It's not possible to have it work any other way that I can think as Xibo needs to write files in to that structure to run.

If you'd like to contribute some pseudo code of how you think it should work I'd be delighted to look them over and take that on board?

Best wishes

Alex

[brodkin]

Alex,

I'd be more than happy to spend some development hours on Xibo, but before I do I want to ensure that I understand your current design. I just a spent a good while looking a little more closely at what is persisted in the volumes and I think there is far more being stored there than is truly necessary.

The approach seems to have been to write the whole CMS to the volume rather than just the stuff that might change from user to user. Is there a reason for this? My approach would be to mount only the library directory and anything else that is modified on disk by xibo (is there anything?). If we modified the Dockerfile to overwrite web/settings.php with one that uses environment variables there would be no need to persist that file any longer. Thoughts?

[alexharrington]

You need to cater for users writing in to the theme directory their own theme.

They will also generate in 1.7 css and Javascript files in the CMS codebase which are for CkEditor to use.

They may also add files to modules, or indeed pretty well anywhere if they have custom modules to run.

The image is also used for development and continuous integration testing where we need to be able to pull in a git checkout of the CMS code at a particular version.

We want our devs and CI system to be using the exact same environment as production instances - otherwise for us the advantages of Docker are lost.

For those reasons it made most sense to add the files to the volume such that they can be modified and backed up easily from outside by a non-expert who has no knowledge of Docker.

I don't think having the CMS in a hard to edit location is therefore viable for us.

[brodkin]

With regard to modifying the codebase, generally I'd say that anyone doing that kind of work should write their own Dockerfile that uses yours as a base image, but if you're dedicated to the idea that someone hacking the CMS should not have to know Docker then there's not much worth discussing.

As far as the issue of dev environments goes, we generally have our build scripts mount the entire application, as you have here, for development and then remove that volume both locally and in CI to run the tests against a releasable image. If the tests pass then we deploy that image to the registry.

Unless you're willing to reduce the unrestricted access that your users have had historically, I don't think you'll be able to realize the complete benefits of deploying with Docker. The current approach is serviceable, but it causes me to question if you wouldn't be better served by a vagrant image instead of docker containers.

I wish you guys the best of luck, but I don't know if I have anything to offer at the moment. If you want to explore the idea of supporting the modification of themes and such via Docker builds please drop me a line and I think there's some exciting work that can be done.

[dasgarner]

Interesting conversation here - thanks for your time brodkin.

One thing that isn't clear to me at present is how Wordpress manage themes and plugins which are stored on the file system, if that file system is container specific and can be lost over upgrade (or indeed node failure). I appreciate that Wordpress allows these things to be uploaded through the webUI without file system access and that it would be nice if Xibo allowed for that too - but at the same time, they do still end up on the file system - and if that is container specific, then it would be lost over upgrade?

As a side note - dev uses vagrant with a docker-provisioner and never runs the launcher script. It uses the same containers so we have the same base environment, but we map the xibo-cms repository for shared file access all the way through to the docker container. I don't think the launcher concerns apply to dev. I think CI does something similar, right? I.e. sets up the environment to run from the checked out repo in Jenkins, rather than running launcher?

We certainly aren't docker experts (at least I'm definitely not) - always happy to explore better ways of doing things.

[alexharrington]

@dasgarner One thing that isn't clear to me at present is how Wordpress manage themes and plugins which are stored on the file system,

They have a clean separation - so wp-content directory contains user themes/modules/uploaded content I believe.

@dasgarner As a side note - dev uses vagrant with a docker-provisioner and never runs the launcher script. It uses the same containers so we have the same base environment, but we map the xibo-cms repository for shared file access all the way through to the docker container. I don't think the launcher concerns apply to dev. I think CI does something similar, right?

Yes absolutely, but as I understand it, with what @brodkin proposes then there would be no way in the image to map through an entire CMS codebase - only various bits (eg the custom directory, library and maybe theme). The issue I see there is that the theme directory contains a mix of both user content and CMS code, so it's no different to the solution we have now, which is far more flexible. CI is run against every pull request (and potentially for every commit in every pull request) so building release archives ahead of each of those tests is impractical. It needs to simply run against a git clone as it does now.

I anticipate a fairly significant opposition to using Docker from our community - at least from those who aren't invested in it already. I don't see that making simple modification to the CMS code require that they can author Docker images is going to make them more likely to embrace the solution - even if it's technically the most correct way to do things. Similarly, we often want people to make a one line change to the CMS code when debugging an issue, and that becomes magnitudes harder to achieve if the CMS code is locked away inside the container.

I totally understand why you would want that in an ideal world, but sadly that's not what we have to work with.

[dasgarner]

I've grasped it now - we'd be removing VOLUME /var/www/xibo in favour of selecting the 3 places that users may write files, which in 1.8 are:

• /custom
• /library
• /theme

Which means that development and patches would require the user to switch to a new container which exposed the full CMS files as we currently do.

/theme does contain CMS core code and we would therefore be looking at separating that out, perhaps by flattening /theme/default into /web and having a /web/custom folder which only contained theme folders.

I certainly do understand the sentiments behind this, and I am not at all adverse to the restructure of that folder - perhaps we can consider that for 1.9, which will already touch that structure anyway.

[brodkin]

@dasgarner You're really close to the full vision here. You'd mount the three volumes that you mentioned, with separation of the custom components into their own directories. That will allow for basic customization for anybody.

I think the next step is adding some ONBUILD directives to the Dockerfile to make building a custom image easier. For instance it could be setup that if a user runs docker build . in a directory with a theme directory, the directory will automagically be placed at /web/usertheme and the user needs to know only how to build, push, and pull containers--they never have to deal with commits or custom Dockerfiles.

For more advanced modifications, advanced Dockerfiles could be constructed to allow for any modification to any file in any directory. Locking down the container won't prevent modification, it'll just require users to interact with Docker a little more.

[alexharrington]

.. which then falls on us to offer support for that - vs a trivial in place modification, which then persists across container rebuilds, container failures etc. I see the vision, but I don't see the advantages personally. What am I missing?

As I understand it you're talking about putting the theme in to the container itself - yet the theme is user data, which I was under the impression should be persisted on an external volume. I don't understand why it's desirable to have to educate the whole userbase on how to build docker images and push them to Dockerhub so that they can apply their own theme - which they may prefer to keep private?

Why not keep the theme in a volume, where they have direct control over it, and updating it is trivial?

[brodkin]

@alexharrington, I don't think that you're wrong, but hopefully I can offer some perspective...

• You mentioned persisting only the theme, and that would actually be preferable to what we're doing now. That said, I do not think it's necessary to keep the theme on disk. The theme is not user data until they modify it, which is not an integral part of using Xibo. Just keep the current version of the theme in the container so that it always stays current with the version of Xibo in use. This prevents you from having to question if a user modification caused a given problem. How do they modify things? read on...
• @dasgarner actually provided the exact solution that I was going to propose for trivial theme modifications. If a /web/custom directory is mounted as a volume then users can place a custom theme there without needing to write Xibo CMS to disk. While you guys would know better than me, I'm guessing that this method addresses the concerns of 98% of your users.
• As far as use of Docker goes, I think it should be supported but not required. I can tell you right now that if I am going to use the software as a vendor I would have a much easier time maintaining a docker image with my modifications than trying to use what's currently being served up. Under the current system all of my modifications are wiped out during upgrade--that's a headache. Imagine instead this Dockerfile:

FROM xibosignage/xibo-cms:latest

ADD lib lib
ADD views/login.twig views/login.twig

While this is a pretty basic example, I can trigger an automatic build of my Xibo container every time that you guys push an update. Then I just pull the latest version of my image and my modifications have already been applied with a new version. BTW, not all Docker images are public, and Docker Hub is not the only registry.

Ultimately, here is what using Docker in the more conventional way gets you:

• Docker containers are lightweight, scalable, disposable, and predictable. Right now we're hitting the first two marks, but not the latter two. Disposing of a container currently does not dispose of the application, just the environment. Similarly, we've introduced unpredictability into a predictable environment by allowing users to modify the entire application. Support seems to be a big issue for you guys... Imagine a world in which you could pull down a clean copy of a container, mount the user's data only, and then see if their problem persists. If it does, it's a configuration error or a bug; if it doesn't, it's something they did to modify the application. Isolating issues becomes a breeze. The application running on every server becomes the same down to the smallest permission--no more guessing.
• Users can upgrade or downgrade freely. Imagine a user upgrades to 1.8.4 and then realizes that a bug was introduced in a key component that they need for their organization. Under the current system they have to sit and wait because their application has been overwritten and there is no way to go back to a prior version. Instead, with separation of the user data and the application, they could just pull the previous tag and everything goes back to being happy.
• As Docker becomes increasingly popular, you may find that the users who have the largest hacking aspirations want to build their own containers that do more. Starting from a place of Docker best practices will support that future growth when it's ready to happen.

[alexharrington]

Thanks for taking the time to explain all of that. It really is appreciated.

So say the dev team make a change that gives a clean separation for the user theme directory, then how do we go about using the same container for dev/CI where we absolutely do need to have the container run the CMS from a volume rather than from inside the container itself?

Also I just wanted to pickup one item:

Users can upgrade or downgrade freely. Imagine a user upgrades to 1.8.4 and then realizes that a bug was introduced in a key component that they need for their organization. Under the current system they have to sit and wait because their application has been overwritten and there is no way to go back to a prior version. Instead, with separation of the user data and the application, they could just pull the previous tag and everything goes back to being happy.

Unfortunately, Xibo CMS database upgrades are a one-way-street, so they'd need to pull the old tag, and then restore their database backup (assuming they took one, since I think you're suggesting that we wouldn't be running any kind of management of that process now). If they didn't restore the database, they'd be looking at a version incompatibility screen.

[brodkin]

@alexharrington, as for your first question regarding the mounting of volumes, we keep a build script in our root directory that does all of that for us. We have built-in commands for dev mode, in which the full application is mounted, and then we have a full-run mode. The the full run builds the containers, runs them in docker, and then runs the test suite against them. The build system we use for this purpose is Robo, in part due to its Docker support, but it could also be accomplished with a bash script if you prefer.

As for the second part, based on your user base I don't think it's wise to give up all management of the upgrade process. It's perfectly acceptable to mount a backups volume, and write to that directly anytime a database upgrade has been executed. I personally would prefer to dump the database from PHP only when there are schema changes, but checking for a version difference via the entrypoint is nearly as good. Naturally the schema has to match, so I may have oversimplified the downgrade process. That said, if the schema does not change it would work as explained.

I'm still not sure what's right for you guys. What I've laid out is the traditional DevOps approach to the matter, but you guys certainly know what's best for your users.

[alexharrington]

@brodkin we're using Jenkins for CI. At the moment that manages pulling and starting the container, running the tests, and then disposing of it afterwards. What I'm unclear about is how it will be able to do that without building a separate container just for that one test? The reason being we're then not testing on the same container we'll push to users for release.

[brodkin]

In our case we push directly from CI after a successful build and test cycle. On Mon, Jun 20, 2016 at 23:54 Alex Harrington notifications@github.com wrote:

@brodkin https://github.com/brodkin we're using Jenkins for CI. At the moment that manages pulling and starting the container, running the tests, and then disposing of it afterwards. What I'm unclear about is how it will be able to do that without building a separate container just for that one test? The reason being we're then not testing on the same container we'll push to users for release.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/xibosignage/xibo/issues/816#issuecomment-227355744, or mute the thread https://github.com/notifications/unsubscribe/AAOcFEy9y1cffghpxHBlXvLeUmZINe7Hks5qN4q4gaJpZM4I43ki .

Brodkin CyberArts http://brodkinca.com | 310.220.0590 DESIGN. DEVELOPMENT. CONSULTING.

Information contained in this email or any attachment may be of a confidential nature which should not be disclosed to, copied or used by anyone other than the addressee. If you receive this email in error, please delete the email from your computer. Do not post anything in this email to any online forum without express permission of the sender. No pixels were harmed in the making of this disclaimer.

[alexharrington]

I don't think that's suited to our workflow really. We test every commit on a PR. Nobody would want to actually run one of those versions.

Is there no way in Docker such that I can define volumes like

/var/www/xibo/library /var/www/xibo/custom /var/www/xibo/web/custom

and also in the same container

/var/www/xibo

Such that I can either expose just the spots most users will need, or the entire CMS tree?

[brodkin]

I'm not sure how your CI is configured, but we only push off of successful builds on the master branch.

As for the Volumes, just to be clear the VOLUME instruction in a Dockerfile does not in any way control or restrict where volumes can be mounted. A single file can be mounted as a volume via the CLI if you like. For your purposes you'd probably want to only define volumes that need to be seeded with configuration files or directory structures. Read the documentation for more info. The /var/www/xibo directory would be mounted via the CLI for development only.

[alexharrington]

OK thanks @brodkin

I've taken all that on board and we will have a discussion internally ahead of the 1.8 beta release.

[brodkin]

Np. Let me know if I can help!

[alexharrington]

Dan added support for user themes going in to web/theme/custom folder today, so I've made a stab at the suggestions you've made.

I'm sure it's not as clean as you would like - and it's untested at present - but it's the best compromise I can find between what "best practices" are, and what we actually need this thing to do. What it's done is move a load of complexity from one place to another.

If you have time to take a look and see what you think that would be great.

https://github.com/xibosignage/xibo-docker/pull/26/files