Dockerize

[jgonggrijp]

@BeritJanssen I would like to be involved in this, as I already have a vision of how to approach it. Perhaps discuss on Signal/Teams?

[BeritJanssen]

On the branch feature/dockerize, there's a first version. This should build, but so far I can only access the backend, and not the frontend via the browser.

TO DOs:

• [x] fill in cookiecutter variables for the name of the containers, database settings etc.
• [x] figure out how to persist data using volumes
• [x] for production use, add a server (gunicorn / nginx?)

I followed this tutorial for the Django/Postgres part, and this one for Angular.

[jgonggrijp]

• [ ] for production use, add a server (gunicorn / nginx?)

I was thinking to do this in development mode, too. I suggest an nginx server that reverse proxies to both backend and frontend.

Development mode:

• backend container
  • runs the Django development server, for example on port 8000
  • livereload for the backend code
  • watching backend unittests
• frontend container
  • runs the Angular development server, for example on port 4200
  • livereload for the frontend code (on a different port than the backend livereload)
  • watching frontend unittests
• nginx container
  • runs the developer-facing web server, for example on port 5000
  • reverse proxies (configurable) selected prefixes such as /static and /api to the backend
  • reverse proxies all other requests to the frontend

Production mode:

• backend container
  • runs python manage.py collectstatic
  • runs the Django application with gunicorn, still on the same port
• frontend container
  • builds the frontend assets
  • still responds to requests on the same port, but now only to serve HTML pages. This could be either a simple connect server that always returns the same index.html (which is enough to implement at first), or if we ever need to be so fancy, an Angular Universal server that pre-renders the HTML for the given path.
• nginx container
  • runs the user-facing web server, still on the same port
  • handles plain file requests on prefixes like /static and /media directly (web servers like nginx have to option to search in multiple directories for a given path, so the build outputs of the backend and frontend don't need to be pooled in a single directory for this purpose. However, the nginx container does need to access them. Can that be done?)
  • reverse proxies (configurable) selected prefixes such as /api to the backend
  • reverse proxies all other requests (i.e., html pages) to the frontend

Above, I left out the functional test suite, which should probably run on yet another container. However, this container has no difference between production and development and does not need to be part of the docker composition. It can test the application on any web address, but if addressing the local docker composition, it should interface with the nginx server just like a regular user.

Dockerization with an approach like the above does have some drawbacks compared to the undockerized situation. Most importantly, since frontend html pages are served directly from the frontend instead of being routed through the backend, session management and authorization are not as straightforward. A simple, but somewhat ugly solution is to work around this by issueing additional requests from the frontend to the backend. A cleaner, but trickier solution is to handle this from the reverse proxy server instead. When forwarding a request to the frontend, it can send an additional request to the backend in parallel to handle session management and then combine the information from both sources when sending the response to the client. Since both variants require that the backend has some kind of dedicated endpoint for session manipulation, starting with the ugly simple solution can be a first step towards the clean solution.

Of course, I do think that the benefits outweigh the drawbacks:

• even better separation between backend and frontend
• fewer differences between production mode and development mode
• local production mode is exactly the same, instead of kind-of the same, as deployment
• deploying on a backoffice server is just setting a single reverse proxy in Apache + git clone + docker compose up, i.e., it is the same for all of our applications and it can be done entirely by backoffice
• full control over the web server that handles requests to our application, without backoffice needing to give us any special permissions

[BeritJanssen]

@jgonggrijp , thanks for lining out your vision for this! Why not run the nginx server on the same container as the frontend? That way we wouldn't need any logic to decide whether the backend or the frontend should handle a request, right?

[jgonggrijp]

Why not run the nginx server on the same container as the frontend?

Because that wouldn't be modular, i.e., it wouldn't fit nicely with #2. In my opinion, the frontend should only be responsible for frontend stuff, and if you create a project without a backend, you don't need the additional middlewoman that nginx is meant to provide.

That way we wouldn't need any logic to decide whether the backend or the frontend should handle a request, right?

I don't see that. Surely we still want to send some requests to the backend container, regardless of what container nginx is run from, and we still need to decide whether to do that or not based on the request path?

[alexhebing]

Hey, great that you're working on / thinking about this! Life should be a lot easier, and at least more homogeneous (as far as moving from development to production is concerned), if all of this is implemented. Nice! Two questions, just because I am curious:

1) Is there any particular reason why we'd use nginx over Apache? It introduces yet another new application, presumably (I am not familiar with it at all) with its own set of things-you-need-to-know(-but-don't-if-you're-new-to-it). Given that there will probably still be older applications on regular servers as well, I'm pretty sure Apache will be around for quite some time, i.e. is there a good reason to introduce nginx next to it?.

2) Does having three docker containers imply three terminals in which we have to start them separately? If not, where does (terminal) output end up?

[jgonggrijp]

Good questions, Alex. Nginx is much more lightweight than Apache. Having two Apache instances on a single server is quite undesirable (especially with the added overhead of three docker containers) while having one Apache instance and one or more Nginx instances is no problem at all. The Nginx developers even encourage you to run it next to other web servers.

Just to avoid any confusion in this regard, Backoffice will still deploy with Apache. Nginx would act as a gateway between Apache and our application.

The entire constellation of docker containers can be started at once with docker compose up, in which case I think all the output will go to a single terminal (@BeritJanssen please correct me if I'm wrong). However, you can also start each container separately if you want.

[BeritJanssen]

Currently, the branch feature/dockerize is using base python:3.6-alpine. I just found this blogpost which states that even though alpine leads to smaller images and faster build times for other environments, such as Go, for Python, it increases build time and image size. Indeed, if you use python:3.6, this will default to debian-buster. So I will update the Dockerfile of the backend to user buster(-slim) instead.

[BeritJanssen]

@jgonggrijp : do you have suggestions of how to handle static files from the frontend, such as logos? I'm a bit stuck with that.

[jgonggrijp]

@BeritJanssen I'll take a look at it today.

[jgonggrijp]

@BeritJanssen See #20. Playing with this, I identified a bunch of things that I think need to be done next:

• [x] Adapt backend and frontend configuration to the new situation (backend doesn't actually work right now).
• [x] Use bind volumes so we don't need to rebuild the entire composition on every code change.
• [x] Re-enable code watching and autoreload (.
• [ ] Make a separation between development and production modes again.
• [ ] Skip collectstatic in development mode.
• [ ] Implement a solution for CSRF handling.

Needless to say, I can help with these and future steps.

Somewhat further down the line, I think it might be nice to just upload our finished images somewhere for deployment, so no building needs to take place on the server.

[BeritJanssen]

Strange that you report backend isn't working. In my local versions of the cookiecutter, this works (both for an empty test app and embedded in khatt).

[jgonggrijp]

I tried visiting localhost:80/api/ with an empty test app. It errored out on the ground that backend is not in the allowed hosts. It's not really surprising, given that the settings.py hasn't been adjusted to the new situation yet. No worry though, it will be easy to fix.

[BeritJanssen]

Alright, yes, indeed, I fixed that in the khatt repo. Drawback of testing in one repo and updating another at the same time... Will push that.

[BeritJanssen]

Okay, some further steps. The backend works, even though static files (css) from the backend are not accessible yet (noticeable when navigating to localhost/api), and the paths to access rest-framework resources are not available: backend:8000/api/something/ doesn't work but localhost/api/something/ does. This is probably to be fixed in the nginx.conf.

I made bind mounts out of the backend and frontend code, which also takes care of the livereload step (need to reload the browser after backend changes, but this is the case for local development without Docker, too).

[BeritJanssen]

Some progress: accessing static files of the backend (style sheets etc.) now works, using collectstatic. The directory of static files is shared between the backend and nginx container using a named volume.

The proxy_pass config for 'api' is also set (this should still become configurable to another backend app name of choice).