posts/docker-compose-django-postgres-nginx/

[utterances-bot]

Docker Compose with NginX, Django, Gunicorn and multiple Postgres databases - pawamoy's website

Findings, thoughts, tutorials, work. Pieces of my mind!

https://pawamoy.github.io/posts/docker-compose-django-postgres-nginx/

[pawamoy]

Original author: Maxim Dunavicher @maximdunavicher
Original date: 2018-07-28T07:05:44Z

Amazing blog, helped me a lot.
Thank you!

[pawamoy]

Original date: 2018-07-28T14:22:03Z

You're welcome, glad I could help :) !

[pawamoy]

Original author: Maxim Dunavicher @maximdunavicher
Original date: 2018-07-29T21:21:53Z

Btw, do you work with pycharm? it seems I can't use the debug command because nginx crashes immidiately.
If you could spare the time to take a look at:
https://stackoverflow.com/q...
here is the full project btw:
https://github.com/maximdun...

I would probably owe you my sanity...

[pawamoy]

Original date: 2018-07-30T11:17:15Z

So, for potential readers, here are some solutions:

- add "nginx" as a dependency of "djangoapp" in docker-compose.yml (so pycharm also runs nginx when specifying service as "djangoapp")

- bind port: 8000:8000 for the djangoapp service (you don't need nginx to debug)

- in pycharm professional edition, you can leave blank the service part of a docker-compose configuration, and pycharm will run "docker-compose up -d", meaning nginx will also be started

[pawamoy]

Original author: Anon
Original date: 2018-08-01T16:16:39Z

I am trying to use a Unix socket instead of a TCP/IP port for the communication between the Gunicorn server and NGINX. I have tried the above post, https://stackoverflow.com/q..., but to no avail. Do you know how to expose the Unix socket from the djangoapp container to the NGINX container?

[pawamoy]

Original date: 2018-08-01T19:33:29Z

Since both djangoapp and nginx services need access to the socket, you need to create the socket into a named volume and bind it to both - if you followed this post you should know how to declare a new volume and bind it to djangoapp and nginx services :)

Let say you want your socket to be located at /run/gunicorn.sock:
- bind the volume to /run in both services
- in Dockerfile, gunicorn would be run with the --bind unix:/run/gunicorn.sock option
- in nginx config file, you would declare your upstream djangoapp server with "server unix:/run/gunicorn.sock"

[pawamoy]

Original author: Anon
Original date: 2018-08-01T20:26:08Z

Thanks for the help and blog :)

[pawamoy]

Original author: Ronaldo Mata @ronaldo_mata
Original date: 2018-08-14T13:51:13Z

I have this problem: "could not translate host name "database1" to address: Name or service not known" can you help me?

[pawamoy]

Original date: 2018-08-14T18:41:52Z

It may be because the database container was not ready before the djangoapp or nginx one. Just try to run your command again. The database will already be initialized in the named volume so it should start faster and be reachable from django/nginx.

[pawamoy]

Original author: Ronaldo Mata @ronaldo_mata
Original date: 2018-08-14T22:40:07Z

Done! Thanks. This post is great!

[pawamoy]

Original author: Alexander Lebedev
Original date: 2018-08-28T11:14:06Z

Thank you for the post! It was really helpful!

Could you please explain why there is a 8000:80 port set up for nginx in `docker-compose` file. This set up didn't work out for me while I was deploying to AWS. I changed to 80:80 and it all worked.

[pawamoy]

Original date: 2018-08-28T13:24:07Z

Ah! It's because I wrote this post for a development environment, not a production one! Indeed in production you must use 80:80. I'll add this as a note. Thanks for your comment :)

[pawamoy]

Original author: Rommel
Original date: 2018-09-18T16:56:18Z

Hi! I got really confused on how to run this on aws without typing the port. I got an error when I change the port of nginx in docker-compose.yaml to 80:80
Thanks!

ERROR: for nginx  Cannot start service nginx: driver failed programming external connectivity on endpoint container_name (0c5c38f6eb6bfa6b6fb775567292a78bf762a977a7cb9890e2933eb272be24a0): Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use

[pawamoy]

Original date: 2018-09-20T18:23:23Z

Hmmm? I suspect you have another program on your AWS listening to port 80. You can check that with a command like sudo lsof -i -P -n | grep LISTEN.

[pawamoy]

Original author: Mihail
Original date: 2018-10-14T15:35:00Z

Hi! When serving static files, more than 1 mb, nginx sends 403. on smaller files everything ok. how can i fix it?

[pawamoy]

Original date: 2018-10-14T18:50:50Z

Hi! I think you can use the "client_max_body_size 100m;" option in NginX config file. https://nginx.org/en/docs/h...

[pawamoy]

Original author: Artem Polikarpov @artem_polikarpov
Original date: 2018-10-20T07:22:44Z

Hi! Everything goes well including "Attaching to scripts_djangoapp_1, scripts_nginx_1". Actually, this step also goes well but then it fails on the very last step with an error: can't chdir to 'mysite', where 'mysite' is my project name used instead of 'help.'
So, the issue happens in the last string of dockerfile: CMD ["gunicorn", "--chdir", "mysite", "--bind", ":8000", "mysite.wsgi:application"].
How could it be fixed?

Note that if I create a separate container simply with docker build djangoapp_image and then run it with any command like docker run -t -i -p 8000:8000 djangoapp_image then everything goes well.

My aim is to make static files working. I use all the code except for the code for databases (since there are no dbs in my app). Therefore I don't use migration command (but when I tried to migrate data it also failed with an error).

It may be also important to add that I use Docker-tools (for Windows 7) and also tried the following command (resulting with the same error: can't chdir to mysite): API_URL=http://192.168.99.100:8000 docker-compose up

Thanks.

[pawamoy]

Original author: Matej Gazda @matejgazda
Original date: 2018-10-29T21:47:26Z

$ docker-compose build
Couldn't connect to Docker daemon at http+docker://localhost - is it running?

Followed your steps and was left with that. Any help?

Solved - no issue with your code.

[pawamoy]

Original date: 2018-11-05T16:58:48Z

Dumb question but are you sure the mysite directory exists, and that you run the command in its parent directory? It could also be a permission problem, though I'm not familiar with permissions on Windows.

[pawamoy]

Original date: 2018-11-05T16:59:48Z

Hi, sorry for the delay, and glad you solved your issue :)

[pawamoy]

Original author: Artem Polikarpov @artem_polikarpov
Original date: 2018-11-05T21:45:03Z

Yes, sure, I have mysite in mysite. My tree is as follows: requirements.txt docker-compose.yml Dockerfile gunicorn.exe mysite ___mysite ___myappname (here are all files and folders for my application) ___manage.py And I run commands from the folder with files including Dockerfile, docker-compose.yml, etc.
And as I wrote before, if I don't use docker-compose.yml and run the following commands, then I get a good working container with running gunicorn and my application (but w/o nginx, of course):
docker build djangoapp_image and then run it with any command like docker run -t -i -p 8000:8000 djangoapp_image

So far, I could manage static files with dj-static module but still hope to make your code running as well since it allows also running a database and anything else if needed. In a month I'll get a new computer with Windows 10 Enterprise, so I'll be able using Docker for Windows in Hyper-V (instead of legacy docker-tools), and then I'll try everything again.
I'll tell you then if it will work.
Thanks for your answer!

[pawamoy]

Original author: Outis
Original date: 2018-11-09T14:05:21Z

I don't know if it's a typo or just me, but for me it wouldn't work until I changed the database1:env_file value to ./config/db/database1_env.

[pawamoy]

Original author: Jameson
Original date: 2018-11-10T22:45:54Z

Pawamoy you're a hero! Thanks a lot, seriously.

1. I'm new to all of this and my question might be stupid, but there is something that I can't seem to wrap my head around. So, I've installed django_debug_toolbar (it's in my pipfile & pipfile.lock), and to get it to work inside docker: I have to jump into the container (right after creating it) using /bin/bash and manually run "python manage.py collectstatic". If I don't do that, debug_toolbar's static files will not be found in the djangoapp/static folder. How come? I would've thought that the collectstatic command we run in our Dockerfile should've already taken care of that. What am I missing?

P.S. The initial collectstatic (in our Dockerfile) is obviously working because there are files in djangoapp/static, just nothing belonging to the debug_toolbar app.

2. Shouldn't we run apt-get update && apt-get upgrade in our setup? Is it something not usually done in docker setups?

[pawamoy]

Original date: 2018-11-12T15:43:10Z

Oh, interesting, thanks. When you say "it wouldn't work", did you have any error messages that could be helpful to understand what was not working?

[pawamoy]

Original date: 2018-11-12T15:51:13Z

A hero, damn :D ! I should write more of these *o* !! Thanks a lot!

So, I guess you correctly added debug_toolbar to your INSTALLED_APPS settings. Now, I remember having issues when updating the static files. Since the dockerfile steps are cached, sometimes rebuilding the image would not update the staticfiles. Sometimes I had to delete the image AND the static volume and rebuild everything to get it to work.

Please let me see in other projects how I dealt with this issue, and hopefully I will come back with a useful answer :) I also used debug_toolbar in another project so I might have additional useful info.

About apt-get update / upgrade, yes, I think it's usually a good idea to run these. However here we don't install even one program through apt-get, so I did not include it, but feel free to add it yourself :) I'm gonna check if it's better for security reasons to update/upgrade and update the post if needed.

Thank you again Jameson!

[pawamoy]

Original author: Isaac T Chikutukutu @isaactchikutukutu
Original date: 2018-11-25T17:36:43Z

Thanks a lot !!!

You've boiled down all the myth about docker deployment & explained everything in a clean & easy to follow format.

However I had some problems with static & media files (Django 2.1 on Python 3.6) but circumvented them using google cloud & this package ( https://django-storages.rea... ). The whole frustration ended up being a blessing in diguise, my site is now blazing fast.

Thanks again

[pawamoy]

Original author: Julien Nuellas @juliennuellas
Original date: 2018-12-04T08:36:09Z

Amazing job!! It helped me a lot.
On my side, I decided to run my commands directly in the docker-compose.
Is there a preference on your side?

[pawamoy]

Original author: Isaac T Chikutukutu @isaactchikutukutu
Original date: 2018-12-07T16:29:00Z

Hie, thanks for the awesome tutorial it cleared the mist for me.

Can you also do another one showing how we can connect Django with Celery using Redis or Rabbit MQ in Docker. I have a setup that's not functioning properly when I use docker but its working perfectly well on my development machine when I'm not using docker.

An added bonus would be also including Celery flower for monitoring my tasks in the docker environment.

Thanks !!

[pawamoy]

Original author: Isaac T Chikutukutu @isaactchikutukutu
Original date: 2018-12-07T16:57:09Z

Fix: I manually ran

sudo docker-compose run --rm djangoapp /bin/bash -c "cd hello; ./manage.py collectstatic"

and removed the command from the Dockerfile, everything worked perfectly

[pawamoy]

Original author: Moritz Gnisia @moritzgnisia
Original date: 2018-12-25T16:45:26Z

Hey Pawamoy, i just would like to thank you for this really nice introduction :) however i had some problems with the static files, the problem was that the volumes would just not update after changing some static files. They would just use the initial configuration, if you might have this problem in the future as well, i just made a small repo with a minimal configuration for a django project and short introduction with the static files: https://github.com/mgnisia/... I hope the quotation is correct of you work in this work, if not please tell me to correct it! Merry Christmas and greetings from germany!

[pawamoy]

Original author: arsenalacid
Original date: 2019-01-06T00:00:38Z

Many thanks for the article it was a great help to me. I can write to static files in docker web app container and the changes are reflected in the code but my local copy of the code doesn't change. For example inside the docker container in static folder I change in home.css some text font to 30 px then I can see the font has increased in size and I can save changes in my docker copy but my local copy outside the docker container it still stays at 20px.

[pawamoy]

Original author: Ryan
Original date: 2019-01-06T07:57:02Z

hey pawamoy, thank you a bunch for the tutorial!
I was following along, but I then wanted to try setting up with Digital Ocean using Caddy as the web server instead. I'm having trouble configuring Docker, with Docker compose...

On DigitalOcean I have DNS setup, so that the IP address of my droplet points to example.com
So the Docker image for Caddy I'm using is the abiosoft one, and I have tried every combination of my domain, ip address, the docker container ip, but cannot get the site to load through Docker.

How would I configure like this tutorial using example.com domain and Docker?

[pawamoy]

Original author: Gajendra Ambi @gajendraambi
Original date: 2019-01-09T09:19:17Z

Please try to include gunicorn with this. It might be unwise to run this in production with django development server.

[pawamoy]

Original author: Steven
Original date: 2019-01-24T10:14:44Z

How do we install new python modules directly inside docker image?

[pawamoy]

Original date: 2019-01-24T13:19:00Z

Do you mean you want to add python modules in your already built image? I'm not used to do this, but I think you can instantiate it (create a container of this image), run a shell in it, add your python modules, and then commit the changes. See https://docs.docker.com/eng... for more info on the docker commit command.

[pawamoy]

Original date: 2019-01-24T13:26:09Z

Hi! Thank you for the kind words :)

Unfortunately I never used Celery nor Rabbit MQ, and just a tiny bit of Redis, so I don't think I could write something on this kind of setup :/ But I'm sure you can find other examples or docker-compose.yml files on GitHub with an advanced search for Celery, Redis and Rabbit MQ.

[pawamoy]

Original date: 2019-01-24T13:27:36Z

Thank you for reporting back :) I really must take the time to come with a fix for this static files problem.

[pawamoy]

Original date: 2019-01-24T13:30:01Z

I'm not sure to understand what you mean by running commands directly in the docker-compose. As for my preference, it will always be for the most elegant solutions! If you feel your way of running commands is easier to understand and maintain, then keep it that way ;) !

[pawamoy]

Original date: 2019-01-24T14:49:27Z

Hi Moritz, sorry for answering so late! Thank you for the link and the credits :) So tell me, did you manage to fix the static files not being updated issue? If so, how did you do it? I took a glance at your repository but couldn't tell if you solved it or not. Thanks again, and greetings from france ;o)

[pawamoy]

Original date: 2019-01-24T14:52:05Z

Hmmm, which one of the static folder inside the container are you modifying? If you change files in the folder pointed to by the STATIC_ROOT, changes will NOT be reflected in your local code, because this directory is stored in the Docker volume, and not bound to your local folder.

[pawamoy]

Original date: 2019-01-24T14:56:52Z

If I understand correctly, I think all you need to do is change this line: https://github.com/pawamoy/... "
server_name localhost;" to "server_name example.com;", and also add "example.com" in Django settings' "ALLOWED_HOSTS" list.

[pawamoy]

Original author: Moritz Gnisia @moritzgnisia
Original date: 2019-01-24T14:57:22Z

Hey Pawamoy, yeah i did the the essential command actually is a docker thing :D so if you update your static files you have to remove the corresponding docker volumes e.g. with `docker volume prune`, there are similar other commands to remove the volumes... Greetings from germany :)

[pawamoy]

Original date: 2019-01-24T15:05:13Z

Ah OK, I'm doing this as well. I think the obvious way to update the static files in the volume would be to manually run 'docker-compose run djangoapp ./manage.py collectstatic' each time. The collection of static files in the Dockerfile could even be removed. Thank you!

[pawamoy]

Original date: 2019-01-24T15:07:44Z

You know what, I think it's the best and most obvious solution. I'm gonna update my post and give you the internet points you deserve (by giving you credits) ;) Thanks a lot!

[pawamoy]

Original author: arsenalacid
Original date: 2019-01-26T14:22:27Z

I see, ok thank you. You are so helpful :)

[pawamoy]

Original date: 2019-02-06T12:27:39Z

Update: so I looked online and it is considered best practice to NOT run apt-get upgrade in Dockerfiles:

- crosbymichael.com/dockerfil... (point 3)
- https://cloudkul.com/blog/d... (point 3 and RUN instruction)
- https://docs.docker.com/dev... (apt-get section)

apt-get update is fine though (use apt-get install in the same RUN instruction to avoid double caching)

[pawamoy]

Original date: 2019-02-11T13:22:15Z

Update 2: I updated the post to remove the collection of static files from the Dockerfile and tell to use it manually when needed. I guess it fixes the debug_toolbar static files problem (without explaining it though).

[pawamoy]

Original author: Mika Gr @mikagr
Original date: 2019-03-05T17:49:00Z

I would like first to thank you for this tutorial, this is the best and more detailed I have read so far!
I managed to run it but I am facing the following error when I try to create a super user:

psycopg2.OperationalError: could not translate host name "database" to address: Unknown host

I launched the "python manage.py createsuperuser" from the virtualenv created by pipenv with the pipfile.

I tried to change database settings in settings.py from "'HOST': 'database'" to "'HOST': 'localhost'" but it does not work neither.

Do you know why and how I can solve this? I am still fairly new to django and web development, so still a long way to go !

[pawamoy]

Original author: Mika Gr @mikagr
Original date: 2019-03-05T18:37:11Z

I found the reason why. We have to use :

docker-compose run --rm djangoapp simulimmo/manage.py createsuperuser

in the same way that we have used it to migrate databases (well explained in your github repo readme)

I let my question in case in help someone.

Thank you again for your work!

[pawamoy]

Original date: 2019-03-07T16:09:39Z

Hello! Thank you for the kind words, it means a lot to me :)

Congrats on fixing your issue by yourself! And thank you for writing this comment and question, I'm sure it will help others indeed :)