Docker Compose

[asbjornu]

If I were to provide a docker-compose.yml file that makes it possible to bootstrap Demozoo by typing docker-compose up on the command line, would you be interested in a PR for that?

[nswaldman]

If the setup has feature parity (or better) as opposed to vagrant, that would definitely be something interesting. Not a lot of people run demozoo for development purposes, so the effort to set that up in Docker was never a high priority, effort/gain wise.

[nswaldman]

In light of https://github.com/demozoo/demozoo/pull/572, it would be a fun exercise to get this conversation rolling again :)

I recall https://github.com/demozoo/demozoo/tree/feature/docker was well on its way to introduce this option, but it requires some updates, refinement and testing still?

[asbjornu]

I've forgotten that this stranded, and even whether it actually stranded at all. Perhaps @gasman remembers the current status?

[nswaldman]

I've locally bumped the branch to postgres 14 and python 3.10 for parity with the Vagrant bump and worked my way through some initial snags:

1. $PWD is a stinker for vanilla Docker installs. The Docker daemon is run as root by default, so unless you alter that, you're going to have to $ sudo PWD=${PWD} docker <whatever> your way through life. A relative path would solve that headache, maybe? If not, maybe document that snag.
2. There is no .docker/.env file and neither an example file nor documentation on how to create a minimal working version exist.
3. I think that's why the db image is boot looping for lack of missing postgres credentials?

Happy to exchange notes here or through the offical Discord :)

[asbjornu]

Thank you for the notes, I now remember being stuck on bootstrapping Postgres as well. I think I got past the credentials, but some initial data or SQL script was required on init, which I was completely unable to get Postgres to load on startup.

[nswaldman]

@gasman I think that is the step where migration.py kicks in, but you are definitely more qualified to have opinions about this :) It would seem reasonable to have that break if the postgres container lacks python and the virtualenv setup that the project relies on?

[asbjornu]

I think I got some way converting the entire thing to a regular .sql script, but I can't remember the details. My problem was getting the .sql script executed by PostgreSQL on startup, IIRC.

[gasman]

@nswaldman Please can you push whatever progress you've made on this to the feature/docker branch?

Once you've got a postgres server up, the most sensible next step would be to fetch the latest public SQL dump from http://data.demozoo.org/demozoo-export.sql.gz and load that in - see https://github.com/demozoo/demozoo/blob/feature/docker/etc/vagrant-provision.sh#L69-L71 for the equivalent steps under Vagrant.

[nswaldman]

I can, but I'm afraid it isn't terribly much -version bumps for images in the Dockerfiles for each container and the creation of a (blank) .env file in the .docker folder got me to the questions/findings mentioned above.

[gasman]

OK, I think I'm getting somewhere... I've now got it successfully starting up the server with a database pulled from the public dump, but with no CSS. You need a .docker/.env file with a few important keys set - I've added a .docker/.env.example you can copy from.

(I suspect that the already-existing top-level .env.example is intended to work here, except that POSTGRES_USER / POSTGRES_PASSWORD are very much required because the db container uses them to set up an account with those credentials. I also notice that the web container is throwing UserWarning: Not reading /home/demozoo/.env - it doesn't exist - but I'm guessing that's harmless, and Docker is shoving those variables into the environment so that Python doesn't actually need to see it as a file. It's definitely picking up SECRET_KEY from there, at least, because the webserver refuses to run without it...)

So, the next hurdle is building the CSS files, which would involve getting a nodeJS v18 environment installed and then running npm i --no-save && npm run build from the project root. I'm guessing that the apt-get incantations we use to install node under Vagrant might work here too - but is that the right thing to do, or should we be doing something different like setting up a dedicated container to run node in? @asbjornu any guidance?

[asbjornu]

We can use multi-staged builds to pull in completely different intermediary containers for building and then throw them all away at the final stage after copying the previous stage's build output with COPY --from=<stage>.

[asbjornu]

591 submitted. I hope it works for you as well as it does for me. :)

[nswaldman]

That looks extremely promising :+1: The only thing that isn't working out of the box is the one-time npm build task, so the css and svg assets were missing.

There are no user accounts by default, that would probably be a useful thing to tack on?

[asbjornu]

Hm, I wonder what kind of local data I have that makes the build work for me. Here's proof that docker compose up --build gives me a working version of Demozoo, with CSS and everything locally:

[nswaldman]

I expected $ docker compose up to be the go-to command, but after I pruned everything (including removing the .docker/db/pgdata) folder and started afresh with the --build flag added for context, it doesn't generate or place the files where demozoo expects them:

demozoo-web  | [29/Jan/2024 15:38:04] "GET /static/css/dz.css HTTP/1.1" 404 1789
demozoo-web  | [29/Jan/2024 15:38:04] "GET /static/images/icons.svg HTTP/1.1" 404 1807

One possibility is that css/dz.css and images/icons.svg are present on your local machine from an earlier local npm build run.

[emoon]

Great stuff! For someone that barley used docker the readme states you should install "Docker Desktop". I wonder if a desktop version is really needed here as I would just like to run it headless over ssh.

[asbjornu]

One possibility is that css/dz.css and images/icons.svg are present on your local machine from an earlier local npm build run.

@nswaldman, that might indeed be the case. I'll do a git clean -xdf to see whether it still works for me. Perhaps @gasman can weigh in on what needs to be copied into the container and where for CSS to work? I mean, npm run build produces something, I'm just not sure where it ends up and what to do with it. 😅

Great stuff! For someone that barley used docker the readme states you should install "Docker Desktop". I wonder if a desktop version is really needed here as I would just like to run it headless over ssh.

@emoon, I guess we can just link to Docker.com and allow people to figure out which version they want to use?

[emoon]

@asbjornu I was just wondering if the desktop version was needed for some specific reason, but I guess that isn't the case then :)

[asbjornu]

@emoon, ah. No, any variant of Docker should be good. :)

[nswaldman]

@nswaldman, that might indeed be the case. I'll do a git clean -xdf to see whether it still works for me. Perhaps @gasman can weigh in on what needs to be copied into the container and where for CSS to work? I mean, npm run build produces something, I'm just not sure where it ends up and what to do with it. 😅

If we'd mount the host directory static_built and let the build process leave its handiwork there it should be good. Having the generated FE assets tucked away on the container would make updating them a bit of a chore :grimacing:

[asbjornu]

@nswaldman, as we set up $PWD as the volume /home/demozoo inside the container, the contents of $PWD/static_built should be modifiable from the host computer. That setup would require npm run build (via node --watch-path ./src, for instance) to be executed locally, though.

By installing Node.js in the runtime stage of the Docker image (instead of as a separate build stage) we might be able to do npm run build in our entrypoint script, somehow. That would lead to a fatter Docker image, but less installation and setup on the host computer.

[nswaldman]

That setup would require npm run build (via node --watch-path ./src, for instance) to be executed locally, though.

In a way that's fine. It is assumed that FE will mostly want to touch css/svg assets and are comfortable running this locally. (I've got to have a look at --watch-path, until then npm run watch covers this :smiley: )

For people with more of a BE focus it would be helpful if npm run build is done at startup so they have a complete setup and not deal with local node shenanigans. As long as node_modules isn't shared between the container and the host like in the link you shared, we should be golden :smiley:

[gasman]

Hi @asbjornu - not sure if you're still waiting on any answers from me or whether you have all the information you need now, but just in case...

I can confirm I'm seeing the same thing as @nswaldman when I run docker-compose up from a clean git checkout - the site is missing CSS.

Perhaps @gasman can weigh in on what needs to be copied into the container and where for CSS to work? I mean, npm run build produces something, I'm just not sure where it ends up and what to do with it. 😅

I can't answer questions of the form "what should the container do" - that's the gap in my knowledge that I'm relying on you to fill. If you can rephrase the question without the word "container", I might be able to help :-) npm run build takes input from src, and generates output files in static_built, which the webserver then needs to access and serve.

A solution that doesn't require developers to have node installed on their host machine is definitely preferable - the aim here is to be able to spin up a working site in one command, after all. (If running node on the host machine is the only way, then that's better than not having a solution at all, of course. Just as long as that's covered in the readme instructions...)

[asbjornu]

@gasman, right. Thanks for the directions! Then I think including npm out of the box in the Docker image is the way to go, so nothing besides Docker is required on the host computer. I figured out that ./static_built/images needs to exist, so I've adjusted that as well as the installation of Node.js and execution of npm run watch (which was already there, I discovered) in the Docker entrypoint. #591 is rebased and updated with these changes.

[gasman]

Still not building CSS for me, I'm afraid... The npm run build / npm run watch & commands are failing with:

demozoo-web  |
demozoo-web  | > demozoo@1.1.1 build
demozoo-web  | > run-p build:*
demozoo-web  |
demozoo-web  | sh: 1: run-p: not found
demozoo-web  | Running the server on http://localhost:8000
demozoo-web  | /home/demozoo/manage.py:9: UserWarning: Not reading /home/demozoo/.env - it doesn't exist.
demozoo-web  |   dotenv.read_dotenv()
demozoo-web  | /home/demozoo/demozoo/celery.py:8: UserWarning: Not reading /home/demozoo/.env - it doesn't exist.
demozoo-web  |   dotenv.read_dotenv(env_file)
demozoo-web  |
demozoo-web  | > demozoo@1.1.1 watch
demozoo-web  | > run-p watch:*
demozoo-web  |
demozoo-web  | sh: 1: run-p: not found
demozoo-web  | /home/demozoo/manage.py:9: UserWarning: Not reading /home/demozoo/.env - it doesn't exist.
demozoo-web  |   dotenv.read_dotenv()
demozoo-web  | /home/demozoo/demozoo/celery.py:8: UserWarning: Not reading /home/demozoo/.env - it doesn't exist.
demozoo-web  |   dotenv.read_dotenv(env_file)
demozoo-web  | Watching for file changes with StatReloader

The run-p executable is apparently provided by the npm-run-all package, and if I run npm install --no-save outside the container, this gets installed in node_modules and npm run build completes correctly. So maybe the contents of node_modules aren't persisting in the container?

I can't see why you would need to create the ./static_built/images directory, since that exists as part of the repo - but happy to leave that step in if it solves a problem.

[asbjornu]

Still not building CSS for me, I'm afraid... The npm run build / npm run watch & commands are failing with:

@gasman, thanks for testing and debugging this with me. If I rm -rf node_modules before running docker compose up --build, it fails for me as well.

The run-p executable is apparently provided by the npm-run-all package, and if I run npm install --no-save outside the container, this gets installed in node_modules and npm run build completes correctly. So maybe the contents of node_modules aren't persisting in the container?

Indeed. I believe the issue is caused by WORKDIR and VOLUME both being set to /home/demozoo, as the VOLUME will effectively erase whatever is copied into the directory during docker build.

I suppose this can be fixed by installing node_modules to a different --prefix, but this is unchartered territory for me, so I will have to do a lot of reading and testing to figure it out. We have two alternatives, though:

1. Reconfigure the VOLUME so it more selectively maps certain directories from $PWD instead of the entire directory. This will require us to move files into subfolders, or leave us with a Docker container that is less dynamic to changes on the host computer. We would have to decide what should warrant a new version of the Docker image (i.e. rebuild) and not. I think this is the cleanest solution, but it's also the one that requires most work.
2. Do npm install within entrypoint. Since entrypoint runs after the VOLUME is mounted, it will be able to create node_modules on the host computer and persist it through reboots of the container. This is not a very clean solution, but I've tested this, and it works.

I can't see why you would need to create the ./static_built/images directory, since that exists as part of the repo - but happy to leave that step in if it solves a problem.

It solves the same problem as doing npm install inside entrypoint: Everything that is going to exist within a VOLUME essentially needs to be created during runtime or be provided by the host computer before the container is started.

[gasman]

Thanks for the further investigation @asbjornu! Maybe it'll be helpful to compare against this other repo from my day job, which is doing something rather similar (a script to set up a development instance for a Django project with front-end assets built with Node)... https://github.com/wagtail/docker-wagtail-develop

In particular, I notice this commit https://github.com/wagtail/docker-wagtail-develop/commit/ac29388bd16291271d497cc6dc5081b4d6a6f4f2 which looks like it might be transitioning from the quick-and-dirty "run npm install on container startup" (the equivalent of your alternative 2) to the preferred solution with npm --prefix / install. There's also a follow-up commit at https://github.com/wagtail/docker-wagtail-develop/commit/a0e731f937b4685f2bad2c4dfe3d1b31853de25f that may or may not be significant...

[asbjornu]

I've done some testing with --prefix and it seems to effectively change the working directory of all npm commands to whatever you assign as the prefix. So when npm --prefix /usr/local/demozoo run build is executed, the following command:

sass ./src/style/layout.scss:./static_built/css/dz.css --style compressed

is effectively translated to:

sass /usr/local/demozoo/src/style/layout.scss:/usr/local/demozoo/static_built/css/dz.css --style compressed

which is not what we want, because src/ resides in /home/demozoo, not /usr/local/demozoo (where only node_modules are supposed to be). Needless to say, the command fails with:

Error reading src/style/layout.scss: no such file or directory

I think the most robust way forward is to be more selective in what we bind as a VOLUME inside the container. Binding the entire root directory to the same folder as the working directory is proving difficult. We should only bind to a VOLUME:

1. What we want to be persistent over several launches of docker compose up (such as the database).
2. Stuff that should change dynamically inside the container when it changes on the host (such as CSS).

Is it difficult to move more of the code into subfolders to simplify such a VOLUME configuration?