Help wanted for making docker easy to run

[revant]

Help wanted for making docker easy to run on:

• TLDR setup, I tried with docker-compose.yml, I'm out of ideas now.
• Linux
• Linux with custom kernels
• Mac
• Windows Pro
• Windows Home
• Virtualbox/VMWare/KVM/XEN/lxd/other VMs available.
• System where port 80 and port 443 are already used, (other complex port mapping may also be done already on host machine) https://github.com/frappe/frappe_docker/issues/354
• Think of more complex scenarios and provide for it in docs. e.g Docker in Linux VM on Windows Server access through VPN
• The installation should be accessible on LAN/WAN/Private network.
• ARM devices

I don't know how to help others with this.

[ceefour]

Hi @revant ,

Unless you see any concerns with this, I figure might as well make frappe_docker repo (and hopefully frappe and erpnext as well) Hacktoberfest-eligible since DigitalOcean changed to an opt-in model this year: https://hacktoberfest.digitalocean.com/details#details

Requires adding a hacktoberfest topic to the repo on GitHub, that should be all that's necessary for this.

[christf]

I am struggling with the setup of erpnext inside docker using docker-compose. based on the env-local file I have created the pods, they are coming up with an .env file based on the local one and when I curl http://myhostname I get an http 404 error.

Edit: Since I am struggling with this, it seems I might be the perfect test pilot for intended improvements. First the issue must be identified though.

[Davincible]

Hi @revant , I am willing to help write out the docker documentation better. The only issues is that I myself am still getting the permisison denied error as has been mentioned throughout the forum and github issues.

I have some ideas on how to improve the documentation, and make it easier to install, but I first need to get my own image working. If you have any suggestions for the permission error I am keen to hear it.

[revant]

Permission error is because the process in container runs as frappe user (non-root user)

The volume mounted by root with root permission causes the problem.

Exec into python container and run

chown -R frappe:frappe /home/frappe/frappe-bench/sites

This step is generally not needed.

[senorsmile]

I have just discovered erpnext and am trying to bring it up to play with.

I am also getting a 404, specifically (with my.domain.com representing my actual sub domain pointed to the machine where this is installed):

my.domain.com does not exist

I've poked around the nginx container, but don't see anything obvious.
I tried poking around the python container. I see proc /proc/1/ running, but not much I can do without being root.

Any idea what is going wrong following the current directions?

[Davincible]

@senorsmile Yeah you need to forward the headers, was also stuck on that problem for way too long. My config:

    location / {
        proxy_pass http://localhost:8060;
        proxy_buffer_size          128k;
        proxy_buffers              4 256k;
        proxy_busy_buffers_size    256k;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_pass_request_headers      on;
    }

[revant]

PR related to nginx config. Check if setting env variables solves your problem.

https://github.com/frappe/frappe_docker/pull/419

[shareef]

Permission error is because the process in container runs as frappe user (non-root user)

The volume mounted by root with root permission causes the problem.

Exec into python container and run

chown -R frappe:frappe /home/frappe/frappe-bench/sites

This step is generally not needed.

I saw the sites are owned by root, when I ran the command I am getting Operation not permitted error. NVM - I got with -u 0 Thanks.

[senorsmile]

@revant

I added UPSTREAM_REAL_IP_ADDRESS='192.168.1.100' to .env. I also had to add - UPSTREAM_REAL_IP_ADDRESS to the docker-compose.yml under

    image: frappe/erpnext-nginx:${ERPNEXT_VERSION}
    restart: on-failure
    environment:
      - FRAPPE_PY=erpnext-python
      - FRAPPE_PY_PORT=8000
      - FRAPPE_SOCKETIO=frappe-socketio
      - SOCKETIO_PORT=9000
      - UPSTREAM_REAL_IP_ADDRESS

I now see that being set in the nginx config.
However, I'm still getting page not found.

@Davincible I assume that you built a custom image with this config?

[shareef]

Folks, I spent good amount of time making the docker-compose stable for dev environment. I am able to compose the containers and dependencies properly so the containers start working with get go. No more config not found, restarts etc...

How can I contribute to this project?

[ddavidebor]

Open a PR! Contributions welcome :-)

On Sun, 14 Mar 2021 at 05:14, shareef @.***> wrote:

Folks, I spent good amount of time making the docker-compose stable for dev environment. I am able to compose the containers and dependencies properly so the containers start working with get go. No more config not found, restarts etc...

How can I contribute to this project?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/frappe/frappe_docker/issues/353#issuecomment-798834619, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACFC37WO4OPY7QGYWPRHFSDTDQZ3ZANCNFSM4RZGUXZQ .

[Davincible]

@shareef oeh that sounds juicy, could you also put it in a public repo?

[Davincible]

@senorsmile I linked the nginx container to my host instead of the python container, as the nginx container is used to serve static content:

  erpnext-nginx:
    image: frappe/erpnext-nginx:${ERPNEXT_VERSION}
    restart: on-failure
    environment:
      - FRAPPE_PY=erpnext-python
      - FRAPPE_PY_PORT=8000
      - FRAPPE_SOCKETIO=frappe-socketio
      - SOCKETIO_PORT=9000
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.erpnext-nginx.rule=Host(${SITES})"
      - "${ENTRYPOINT_LABEL}"
    # - "${CERT_RESOLVER_LABEL}"
      - "traefik.http.services.erpnext-nginx.loadbalancer.server.port=80"
    ports:
      - "8060:80"
    volumes:
      - sites-vol:/var/www/html/sites:rw
      - assets-vol:/assets:rw
      - /var/run/mysqld:/mariadb_socket

And then comment out the Traefik image

[shareef]

@shareef oeh that sounds juicy, could you also put it in a public repo?

Sure, I will cleanup and add to public repo.

[shareef]

@senorsmile I linked the nginx container to my host instead of the python container, as the nginx container is used to serve static content:

  erpnext-nginx:
    image: frappe/erpnext-nginx:${ERPNEXT_VERSION}
    restart: on-failure
    environment:
      - FRAPPE_PY=erpnext-python
      - FRAPPE_PY_PORT=8000
      - FRAPPE_SOCKETIO=frappe-socketio
      - SOCKETIO_PORT=9000
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.erpnext-nginx.rule=Host(${SITES})"
      - "${ENTRYPOINT_LABEL}"
    # - "${CERT_RESOLVER_LABEL}"
      - "traefik.http.services.erpnext-nginx.loadbalancer.server.port=80"
    ports:
      - "8060:80"
    volumes:
      - sites-vol:/var/www/html/sites:rw
      - assets-vol:/assets:rw
      - /var/run/mysqld:/mariadb_socket

And then comment out the Traefik image I wasn't sure also why traefik image was used, need a picture to visualize all the containers and their uses along with volumes. Let me put one and ask for reviews. This is for my own understanding.

[senorsmile]

Thanks @Davincible. I'm now much further along, although I still get errors.

192.168.1.100 does not exist

docker logs does now show corresponding log entries, confirming 404.

NB: other than setting the UPSTREAM_REAL_IP_ADDRESS env var, I have not modified the built in nginx container.

Poking around the nginx configs more to see what I can figure out.

[senorsmile]

Entering into the python container and poking around, I realized that the site name is specific to the domain name chosen. I edited my /etc/hosts to 192.168.1.100 erp.example.com

I am now getting a 500 Internal server error

I added ports for the python container 8081:80 and accessed the container directly. I get the same internal server error.

docker logs for the python container shows

pymysql.err.OperationalError: (1045, "Access denied for user '_22835929e7d52caf'@'172.20.0.2' (using password: YES)")

so, presumably there's a problem with either the mariadb password it's trying to use, and/or the password that it was set up with.

[Davincible]

@senorsmile you need to 1. set a website name in your env file, 2. setup the nginx forward proxy as I showed, 3. link your domain, or edit the /etc/hosts on your host machine if you are not using a real domain. With those steps I can get it to work

[Davincible]

@senorsmile Oh just reading the part about password denied now, yeah that is a wholenother error. I got that too, and tbh I'm not sure how it got resolved, sometimes it works sometimes it doesn't. I do remember manually playing around with the passwords and access permission in the database

[revant]

Common cause of "Access denied for user 'db_user'@'x.x.x.x' is that the database user is created with specific host instead of % wildcard. In case of docker, the ips of containers connecting to the mariadb can vary dynamically and mariadb user needs to have wildcard host.

Either set wildcard host for the user or add a new user with db_user and db_password with wildcard host

check this https://github.com/frappe/frappe_docker/tree/develop/development#fixing-mariadb-issues-after-rebuilding-the-container

[senorsmile]

@revant

I have not rebuilt the container. I've removed all containers and brought them back up over and over. I'm trying to just use what's here and create a patch list so that I (and others) can simply cp env-* .env, customize and docker-compose up.

It's not clear to me what would need to change to account for this.
Is this a change that needs to happen in one of the container builds as is in the repo right now, or something that needs to change in the way that containers are being brought up for the first time?

[Davincible]

@revant Can't the database wildcard be set automatically? Doing it manually is a fix but it very cumbersome, especially for new users

[shareeef]

@revant

I have not rebuilt the container. I've removed all containers and brought them back up over and over. I'm trying to just use what's here and create a patch list so that I (and others) can simply cp env-* .env, customize and docker-compose up.

It's not clear to me what would need to change to account for this. Is this a change that needs to happen in one of the container builds as is in the repo right now, or something that needs to change in the way that containers are being brought up for the first time?

I recently updated the docker compose file and did following: 1) started Maria db first with a health check... 2) adjusted dependencies to start in order when Maria db is healthy... I am able to consistently able to do what you are suggesting....

[shareeef]

I will post my changes tonight

[shareef]

I added healthcheck section to mariadb service: Please adjust start_period accordingly, I kept it 4 mins but you can adjust per your environment.

 mariadb:
    image: mariadb:10.3
    restart: on-failure
    healthcheck:
      test: ["CMD", "mysqladmin" ,"--password=$${MYSQL_ROOT_PASSWORD}","ping", "-h", "localhost"]
      start_period: 240s
      interval: 90s
      timeout: 90s
      retries: 3
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
    volumes:
    ....

I added mariadb dependency on other services like this

erpnext-python:
    image: frappe/erpnext-worker:${ERPNEXT_VERSION}
    restart: on-failure
    environment:
      - MARIADB_HOST=${MARIADB_HOST}
      - REDIS_CACHE=redis-cache:6379
      - REDIS_QUEUE=redis-queue:6379
      - REDIS_SOCKETIO=redis-socketio:6379
      - SOCKETIO_PORT=9000
      - AUTO_MIGRATE=1
    depends_on:
      mariadb:
        condition: service_healthy
      traefik:
        condition: service_started
      redis-cache:
        condition: service_started
      redis-queue:
        condition: service_started
      redis-socketio:
        condition: service_started
    volumes:
      - sites-vol:/home/frappe/frappe-bench/sites:rw
      - assets-vol:/home/frappe/frappe-bench/sites/assets:rw
...

I also changed installation\frappe_mariadb.cnf

[mysqld]
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
innodb-file-format = barracuda
innodb-file-per-table = 1
innodb-large-prefix = 1

[mysql]
default-character-set = utf8mb4

Hope this helps

[revant]

@revant Can't the database wildcard be set automatically? Doing it manually is a fix but it very cumbersome, especially for new users

It is set automatically if things go as expected.

https://github.com/frappe/frappe_docker/blob/5121e2710653c8df513e953a21e8d1ee8d3c2bde/build/common/commands/new.py#L106

[Davincible]

@revant Hmm okay, and when do things not go as expected?

[shareef]

When database is not yet ready....you will see all the issues...

On Mar 19, 2021, at 5:12 AM, David Brouwer @.***> wrote:

 @revant Hmm okay, and when do things not go as expected?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[github-actions[bot]]

This issue has been automatically marked as stale. You have a week to explain why you believe this is an error.

[nileio]

hi all - with the basic bench install using docker , i recieve 404 page not found error. nginx logs dont have anything relevant traefik is running and exposing both 80, and 443 logs look ok no idea anyone actually tried the docker-compose file ?

[revant]

@nileio What are the traefik logs?

[nileio]

@revant just a note that the PWD stack actually works which uses a different docker-compose it looks like. not really sure where the problem is with the existing docker-compose file in the main repo. Did anyone actually tried it ?! I am sure I aint the only one getting 404. I spend few hours then deleted the whole thing , need to do it again to get you any logs. but at least the PWD stack works for me but it is very slow of course. could you confirm that in a clean docker machine , you can get it running using existing docker-compose file?

[revant]

Docker installation is not easy yet. Re opening the issue.

[tejpochiraju]

I have just started trying to build images for a custom app (without ERPNext). If I succeed, I will contribute that to the documentation.

[revant]

I have just started trying to build images for a custom app (without ERPNext). If I succeed, I will contribute that to the documentation.

Refer https://github.com/castlecraft/custom_frappe_docker (with ERPNext)

https://github.com/revant/erpnext_feature_board/tree/main/containers (only Frappe + custom app)

[pratikbin]

We are running ERPnext with frappe_s3_attachment custom app

frappe_s3_attachment: https://github.com/zerodha/frappe-attachments-s3

Dockerfiles(check README): https://github.com/Improwised/frappe_docker

[github-actions[bot]]

This issue has been automatically marked as stale. You have a week to explain why you believe this is an error.

[daraul]

Be nice if this was revisited. I have team members that are not having much luck with erpnext on windows.

[revant]

I'm the only one here. I can't help much. I don't use or plan to use windows/mac. If I re-poen bot will close it anyway. Or the issue will be kept open just for the sake of it. Thats what happened for a year!

Check this if it helps: https://github.com/frappe/frappe_docker/wiki/Troubleshoot#windows-based-installation

Containers works great on linux. They depend on Linux kernel to work.

On windows and mac I think it runs the Linux kernel under VM (WSL derivative or Virtualbox). That means docker also runs under VM. So VM port forwarding, networking and resource allocation is additional overhead to get docker running on non Linux kernels.

EDIT: I added "Help Wanted" label. Nothing really changed. Hacktoberfest is for newbies to get t-shirts. Instead open specific issues with the problems you face. https://dontasktoask.com/

[zwessels]

If I re-poen bot will close it anyway.

maybe having the bot is counter-productive for this repo. doesn't help issues get closed automatically all the time without it being resolved.

at least having the issue stay open indicates to new & existing users that the issue hasn't been resolved

[daraul]

I agree with @zwessels but I also sympathize with @revant. I gave no intention of switching to windows either. I'll do what I can for my team members, and post what I can here. Hopefully that can help whoever comes along in the future.

[saru2020]

I have just started trying to build images for a custom app (without ERPNext). If I succeed, I will contribute that to the documentation.

Refer https://github.com/castlecraft/custom_frappe_docker (with ERPNext)

https://github.com/revant/erpnext_feature_board/tree/main/containers (only Frappe + custom app)

@revant how do I fetch private repo from github?

i tried adding below line in Dockerfile of nginx : /install_app <private app name> https://${GIT_USERNAME}:${GIT_SECRET}@github.com/saru2020/<private app name> main && \, and in the Docker file of worker: install_app <private app name> https://${GIT_USERNAME}:${GIT_SECRET}@github.com/saru2020/<private app name> main and I've put GIT_USERNAME & GIT_SECRET in .env, but the env vars are not being used and hence the github authentication fails.

also, tried adding environment vars for custom-app-erpnext-nginx in docker-compose.yml:

environment:
      - FRAPPE_PY=erpnext-python
      - FRAPPE_PY_PORT=8000
      - FRAPPE_SOCKETIO=frappe-socketio
      - SOCKETIO_PORT=9000
      - GIT_USERNAME=${GIT_USERNAME}
      - GIT_SECRET=${GIT_SECRET}

but still facing the issue.

where should I put these github/repo's username & secret env vars?

note: when I hardcode the username & secret as part of the repo's URL, it works fine.

[revant]

https://discuss.erpnext.com/t/frappe-docker-how-to-build-a-custom-image-with-a-private-repo/84845/2

[saru2020]

thank you @revant, got it working with a few tweaks on top of it.