Trying (unsuccessfully) to install custom apps through the recommended way

kCyborg commented 1 year ago

Hi there, first of all, thanks so much for your work, I really appreciate it, and the documentation muuaaaaaa (a cheff's kiss).

I write because I want to install some apps over the dockerized ERPNext deployment. Following I will try to reproduce my steps:

The custom apps I need are: hrms, payments and non_profit

Create a custom image with the required apps:

I follow the steps, but just to be sure (yes, I has done this several times), I do it manually, so:

I open a file and called it apps.json, where I paste the following:

[
{
      "url": "https://github.com/frappe/payments",
      "branch": "develop"
    },
    {
      "url": "https://github.com/frappe/erpnext",
      "branch": "version-14"
    },
    {
      "url": "https://github.com/frappe/hrms",
      "branch": "develop"
    },
    {
      "url": "https://github.com/frappe/non_profit",
      "branch": "develop"
    }
]

Then I throw the context to a file, but hashed with:

cat apps.json | base64 -w 0 > apps_64.json

And then I save the content of the hashed file with: export APPS_JSON_BASE64=$(base64 -w 0 apps.json)

I tested the integrity of the json file with the jq

I build the custom image with the apps inside with:

docker build \
  --build-arg=FRAPPE_PATH=https://github.com/frappe/frappe \
  --build-arg=FRAPPE_BRANCH=version-14 \
  --build-arg=PYTHON_VERSION=3.10.5 \
  --build-arg=NODE_VERSION=16.18.0 \
  --build-arg=APPS_JSON_BASE64=$APPS_JSON_BASE64 \
  --tag=mycustom_erp:1.0.0 \
  --file=images/custom/Containerfile .

In order to make it easy for me I start up a local docker container registry with:

docker run -d -p 5000:5000 --restart=always --name registry registry:2

And then tag my crafted (with love) image and then upload it to my local registry with:

docker tag mycustom_erp:1.0.0 localhost:5000/mycustom_erp
docker push localhost:5000/mycustom_erp

So far, so good:

Edit the compose.yaml file, so I change it to:

x-customizable-image: &customizable_image
  image: localhost:5000/mycustom_erp
  #pull_policy: never

### This is a snippet, as it's the only thing I need to modify

Follow the steps in the single server example docx (once again, amazin work guys! really neat documentation!!!).

All good so far, but when I try to check for my custom apps I can't find them!!!

So the question(s) is:

Am I missing some steps?
After the installation should I activate the apps somehow or will they activated by default if I use my customized docker image in the compose.yaml file?
Am I missing some extra steps?

Some extra information:

Once the deployment is completed and I create some sites... I access the frontend and backend and I try to find the apps.json file, and there are there indeed and fully loaded with my used apps.json back in the firsts steps of this Issue

frontend

This image is taken from the frontend container The image above shows only a tail, this is made in order to fit into the screen, sorry for the cut

backend

This image is taken from the backend container

revant commented 1 year ago

execute following to install app on site

bench --site site.localhost install-app hrms

can be done with additional opts during site creation, bench new-site ... --install-app=erpnext --install-app=hrms

kCyborg commented 1 year ago

Thanks you so much for the quick answer @revant!!!

Before your answer I tried some different ways to do it: --install-app 'payments ', 'erpnext', --install-app 'payments, erpnext'... but they command line always gave answers like: There is no module name called: payments, erpnext, hahaha, silly me.

I already tested the creation of a new site using your last suggestion, with something like:

docker compose --project-name erpnext-two exec backend \
  bench new-site three.example.com --no-mariadb-socket --mariadb-root-password changeit \
  --install-app erpnext --install-app hrms --install-app payments --install-app non_profit \
  --admin-password changeit

Please, note the extra --install-app passed as arguments...

Aaaanddd....

I got my apps installed!!! Although I could install the non_proffit one :-(

Any idea on how can I achieve it?

Regarding to your first suggestion

bench --site site.localhost install-app hrms

I tried this way to, but once installed the app the container stop working for me. I tried something like the follows:

docker compose --project-name erpnext-one exec backend bench get-app --branch=version-14 payments https://github.com/frappe/payments
bench --site one.example.com install-app payments

But when I go to my browser and try to access the site, it gaves me a 504 error, meaning that somehow I broke the backend container... So the question arise: once the app is installed in an already running site should I somehow restart the backend container? If I do so, the configuration and previously installed apps will be loose? I'm trying deployment using the nicely write documentation here.

Once again, thanks so much for your work... You rock!!!

revant commented 1 year ago

Make sure you use same version/branches for frappe, erpnext, hrms, and other apps wherever available. You are mixing develop, version-14
You cannot build assets in container. bench build will cause problems in production containers.
You cannot bench get-app in container, you need to build it part of image.

https://github.com/frappe/frappe_docker/wiki/Frequently-Asked-Questions#how-to-build-assets

https://github.com/frappe/frappe_docker/wiki/Frequently-Asked-Questions#how-to-install-official-or-custom-apps

azwalzaman commented 1 year ago

Make sure you use same version/branches for frappe, erpnext, hrms, and other apps wherever available. You are mixing develop, version-14

You cannot build assets in container. bench build will cause problems in production containers.

You cannot bench get-app in container, you need to build it part of image.

https://github.com/frappe/frappe_docker/wiki/Frequently-Asked-Questions#how-to-build-assets

https://github.com/frappe/frappe_docker/wiki/Frequently-Asked-Questions#how-to-install-official-or-custom-apps

Thanks for the response. I was just coming in to see why bench build messed up my production site (recreating the containers seemed to fix it)

yalattas commented 1 year ago

Indeed it doesn't work when you add a new app even using the recommended way.

Here is the outcome

munjiz-erp-create-site-1     | Getting hrms
munjiz-erp-create-site-1     | $ git clone https://github.com/frappe/hrms.git   --origin upstream
munjiz-erp-create-site-1     | Cloning into 'hrms'...
munjiz-erp-create-site-1     | Ignoring dependencies of hrms. To install dependencies use --resolve-deps
munjiz-erp-create-site-1     | Installing hrms
munjiz-erp-create-site-1     | $ /home/frappe/frappe-bench/env/bin/python -m pip install --quiet --upgrade -e /home/frappe/frappe-bench/apps/hrms 
munjiz-erp-create-site-1     | $ bench build --app hrms
✔ Application Assets Linked                                            s        node_modules                                                                                
munjiz-erp-create-site-1     | 
munjiz-erp-create-site-1     | 
munjiz-erp-create-site-1     | yarn run v1.22.19
munjiz-erp-create-site-1     | $ node esbuild --production --apps hrms --run-build-command
munjiz-erp-create-site-1     | clean: postcss.plugin was deprecated. Migration guide:
munjiz-erp-create-site-1     | https://evilmartians.com/chronicles/postcss-8-plugin-migration
munjiz-erp-create-site-1     | clean: postcss.plugin was deprecated. Migration guide:
munjiz-erp-create-site-1     | https://evilmartians.com/chronicles/postcss-8-plugin-migration
munjiz-erp-create-site-1     | clean: postcss.plugin was deprecated. Migration guide:
munjiz-erp-create-site-1     | https://evilmartians.com/chronicles/postcss-8-plugin-migration
munjiz-erp-create-site-1     | clean: postcss.plugin was deprecated. Migration guide:
munjiz-erp-create-site-1     | https://evilmartians.com/chronicles/postcss-8-plugin-migration
munjiz-erp-create-site-1     | Browserslist: caniuse-lite is outdated. Please run:
munjiz-erp-create-site-1     |   npx browserslist@latest --update-db
munjiz-erp-create-site-1     |   Why you should do it regularly: https://github.com/browserslist/browserslist#browsers-data-updating
munjiz-erp-create-site-1     | File                                                        Size
munjiz-erp-create-site-1     | 
munjiz-erp-create-site-1     | hrms/dist/js/
munjiz-erp-create-site-1     | ├─ hrms.bundle.MEIOEUYN.js                                  0.90 Kb
munjiz-erp-create-site-1     | └─ performance.bundle.2O6LUNQD.js                           8.55 Kb
munjiz-erp-create-site-1     | 
munjiz-erp-create-site-1     | hrms/dist/css/
munjiz-erp-create-site-1     | └─ hrms.bundle.IBO7PS7A.css                                 1.20 Kb
munjiz-erp-create-site-1     | 
munjiz-erp-create-site-1     | hrms/dist/css-rtl/
munjiz-erp-create-site-1     | └─ hrms.bundle.6M7VB5YS.css                                 1.20 Kb
munjiz-erp-create-site-1     | 
munjiz-erp-create-site-1     |  DONE  Total Build Time: 681.351ms
munjiz-erp-create-site-1     | 
munjiz-erp-create-site-1     | Done in 0.97s.
munjiz-erp-create-site-1     | WARN: restart failed: Couldn't find supervisorctl in PATH

I am using docker-compose and just changed few lines. Where I mount apps.json to new directly and setting environment variable to the same directory.

Here is snap of the changed docker-compose.yaml file

create-site:
    image: frappe/erpnext:v14.23.4
    deploy:
      restart_policy:
        condition: none
    volumes:
      - sites:/home/frappe/frappe-bench/sites
      - logs:/home/frappe/frappe-bench/logs
      - ./erpnext:/home/frappe/frappe-bench/puritec
    environment:
      APPS_JSON_BASE64: $(base64 -w 0 /home/frappe/frappe-bench/puritec/apps.json)
...

Here is the full docker-compose file

version: "3"

services:
  backend:
    image: frappe/erpnext:v14.23.4
    deploy:
      restart_policy:
        condition: on-failure
    volumes:
      - sites:/home/frappe/frappe-bench/sites
      - logs:/home/frappe/frappe-bench/logs

  configurator:
    image: frappe/erpnext:v14.23.4
    deploy:
      restart_policy:
        condition: none
    entrypoint:
      - bash
      - -c
    command:
      - >
        ls -1 apps > sites/apps.txt;
        bench set-config -g db_host $$DB_HOST;
        bench set-config -gp db_port $$DB_PORT;
        bench set-config -g redis_cache "redis://$$REDIS_CACHE";
        bench set-config -g redis_queue "redis://$$REDIS_QUEUE";
        bench set-config -g redis_socketio "redis://$$REDIS_SOCKETIO";
        bench set-config -gp socketio_port $$SOCKETIO_PORT;
    environment:
      DB_HOST: db
      DB_PORT: "3306"
      REDIS_CACHE: redis-cache:6379
      REDIS_QUEUE: redis-queue:6379
      REDIS_SOCKETIO: redis-socketio:6379
      SOCKETIO_PORT: "9000"
    volumes:
      - sites:/home/frappe/frappe-bench/sites
      - logs:/home/frappe/frappe-bench/logs

  create-site:
    image: frappe/erpnext:v14.23.4
    deploy:
      restart_policy:
        condition: none
    volumes:
      - sites:/home/frappe/frappe-bench/sites
      - logs:/home/frappe/frappe-bench/logs
      - ./erpnext:/home/frappe/frappe-bench/puritec
    environment:
      APPS_JSON_BASE64: $(base64 -w 0 /home/frappe/frappe-bench/puritec/apps.json)
    entrypoint:
      - bash
      - -c
    command:
      - >
        wait-for-it -t 120 db:3306;
        wait-for-it -t 120 redis-cache:6379;
        wait-for-it -t 120 redis-queue:6379;
        wait-for-it -t 120 redis-socketio:6379;
        export start=`date +%s`;
        until [[ -n `grep -hs ^ sites/common_site_config.json | jq -r ".db_host // empty"` ]] && \
          [[ -n `grep -hs ^ sites/common_site_config.json | jq -r ".redis_cache // empty"` ]] && \
          [[ -n `grep -hs ^ sites/common_site_config.json | jq -r ".redis_queue // empty"` ]];
        do
          echo "Waiting for sites/common_site_config.json to be created";
          sleep 5;
          if (( `date +%s`-start > 120 )); then
            echo "could not find sites/common_site_config.json with required keys";
            exit 1
          fi
        done;
        echo "sites/common_site_config.json found";
        bench new-site frontend --no-mariadb-socket \
                                --admin-password=admin \
                                --db-root-password=admin \
                                --install-app erpnext \
                                --set-default;
  db:
    image: mariadb:10.6
    healthcheck:
      test: mysqladmin ping -h localhost --password=admin
      interval: 1s
      retries: 15
    deploy:
      restart_policy:
        condition: on-failure
    command:
      - --character-set-server=utf8mb4
      - --collation-server=utf8mb4_unicode_ci
      - --skip-character-set-client-handshake
      - --skip-innodb-read-only-compressed # Temporary fix for MariaDB 10.6
    environment:
      MYSQL_ROOT_PASSWORD: admin
    volumes:
      - db-data:/var/lib/mysql

  frontend:
    image: frappe/erpnext:v14.23.4
    deploy:
      restart_policy:
        condition: on-failure
    command:
      - nginx-entrypoint.sh
    environment:
      BACKEND: backend:8000
      FRAPPE_SITE_NAME_HEADER: frontend
      SOCKETIO: websocket:9000
      UPSTREAM_REAL_IP_ADDRESS: 127.0.0.1
      UPSTREAM_REAL_IP_HEADER: X-Forwarded-For
      UPSTREAM_REAL_IP_RECURSIVE: "off"
      PROXY_READ_TIMOUT: 120
      CLIENT_MAX_BODY_SIZE: 50m
    volumes:
      - sites:/home/frappe/frappe-bench/sites
      - logs:/home/frappe/frappe-bench/logs
    ports:
      - "8080:8080"

  queue-default:
    image: frappe/erpnext:v14.23.4
    deploy:
      restart_policy:
        condition: on-failure
    command:
      - bench
      - worker
      - --queue
      - default
    volumes:
      - sites:/home/frappe/frappe-bench/sites
      - logs:/home/frappe/frappe-bench/logs

  queue-long:
    image: frappe/erpnext:v14.23.4
    deploy:
      restart_policy:
        condition: on-failure
    command:
      - bench
      - worker
      - --queue
      - long
    volumes:
      - sites:/home/frappe/frappe-bench/sites
      - logs:/home/frappe/frappe-bench/logs

  queue-short:
    image: frappe/erpnext:v14.23.4
    deploy:
      restart_policy:
        condition: on-failure
    command:
      - bench
      - worker
      - --queue
      - short
    volumes:
      - sites:/home/frappe/frappe-bench/sites
      - logs:/home/frappe/frappe-bench/logs

  redis-queue:
    image: redis:6.2-alpine
    deploy:
      restart_policy:
        condition: on-failure
    volumes:
      - redis-queue-data:/data

  redis-cache:
    image: redis:6.2-alpine
    deploy:
      restart_policy:
        condition: on-failure
    volumes:
      - redis-cache-data:/data

  redis-socketio:
    image: redis:6.2-alpine
    deploy:
      restart_policy:
        condition: on-failure
    volumes:
      - redis-socketio-data:/data

  scheduler:
    image: frappe/erpnext:v14.23.4
    deploy:
      restart_policy:
        condition: on-failure
    command:
      - bench
      - schedule
    volumes:
      - sites:/home/frappe/frappe-bench/sites
      - logs:/home/frappe/frappe-bench/logs

  websocket:
    image: frappe/erpnext:v14.23.4
    deploy:
      restart_policy:
        condition: on-failure
    command:
      - node
      - /home/frappe/frappe-bench/apps/frappe/socketio.js
    volumes:
      - sites:/home/frappe/frappe-bench/sites
      - logs:/home/frappe/frappe-bench/logs

volumes:
  db-data:
  redis-queue-data:
  redis-cache-data:
  redis-socketio-data:
  sites:
  logs:

apps.json file

path: PROJECT_DIRECTORY_ROOT/erpnext/apps.json is being mounted to /home/frappe/frappe-bench/puritec then an environment variable is pointing to that file APPS_JSON_BASE64: $(base64 -w 0 /home/frappe/frappe-bench/puritec/apps.json)

{
    "erpnext": {
        "url": "https://github.com/frappe/erpnext",
        "branch": "version-14"
    },
    "payments": {
        "url": "https://github.com/frappe/payments",
        "branch": "version-14"
    },
    "hrms": {
        "url": "https://github.com/frappe/hrms",
        "branch": "version-14"
    },
    "ecommerce_integrations": {
        "url": "https://github.com/frappe/ecommerce_integrations",
        "branch": "main"
    }
}

postgres

Secondly, configurator doesn't work whatsoever, I tried it with MySQL/mariadb and postgres as well.

When I used official file as you provided, it worked but not when I changed db to postgres even though I changed a few files and ports every where. Backend was able to connect to db and provision around 725 tables and I could see it in DB. But sockets and frontend breaks for unknown reason.

I am not sure what's the correlation between sockets and DB compatibility. It should be with backend and not model layer nor ORM driver

revant commented 1 year ago

frappe/erpnext:v14.23.4

You cannot add hrms to this image at runtime, you've to build and use custom image.

revant commented 1 year ago

Follow this to build image with custom apps

https://github.com/frappe/frappe_docker/blob/main/docs/custom-apps.md

frappe / frappe_docker