search

zerodha / frappe-attachments-s3

A frappe app to upload file attachments in doctypes to s3.
Other
119 stars 134 forks source link

Cannot run this app in erpnext v12 #36

Closed pratikbin closed 3 years ago

pratikbin commented 3 years ago

Expected behaviour

want to run this app in erpnext deployed on Kubernetes

Actual behaviour

should run without crashloopbackoff in Kubernetes, scheduler pod logs 

config file not created, retry 1
Attempt 1 to connect to mariadb.erpnext.svc.cluster.local:3306
Attempt 1 to connect to frappe-erpnext-redis-queue:12000
Attempt 1 to connect to frappe-erpnext-redis-cache:13000
Attempt 1 to connect to frappe-erpnext-redis-socketio:11000
Connections OK
Starting background scheduler . . .
Traceback (most recent call last):
  File "/home/frappe/frappe-bench/commands/background.py", line 11, in <module>
    main()
  File "/home/frappe/frappe-bench/commands/background.py", line 6, in main
    start_scheduler()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/utils/scheduler.py", line 34, in start_scheduler
    schedule.every(frappe.get_conf().scheduler_tick_interval or 60).seconds.do(enqueue_events_for_all_sites)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 254, in get_conf
    with init_site(site):
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 263, in __enter__
    init(self.site)
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 187, in init
    setup_module_map()
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 1047, in setup_module_map
    for module in get_module_list(app):
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 913, in get_module_list
    return get_file_items(os.path.join(os.path.dirname(get_module(app_name).__file__), "modules.txt"))
  File "/home/frappe/frappe-bench/apps/frappe/frappe/__init__.py", line 871, in get_module
    return importlib.import_module(modulename)
  File "/opt/bitnami/python/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'frappe-attachments-s3'

redis & maridb connections are fine

Step to reproduce

  1. clone this repo and followed these steps
  2. worker image docker file, image on docker hub build it with docker build --no-cache . -t xxxxxx 
    
FROM frappe/erpnext-worker:edge

RUN install_app frappe-attachments-s3 https://github.com/zerodhatech/frappe-attachments-s3

docker build logs 
```log
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM frappe/erpnext-worker:edge
 ---> de2b9eeda132
Step 2/2 : RUN install_app frappe-attachments-s3 https://github.com/zerodhatech/frappe-attachments-s3
 ---> Running in ec1fca2a96b0
Cloning into 'frappe-attachments-s3'...
Obtaining file:///home/frappe/frappe-bench/apps/frappe-attachments-s3
Collecting boto3==1.4.7
  Downloading boto3-1.4.7-py2.py3-none-any.whl (128 kB)
Collecting botocore==1.7.48
  Downloading botocore-1.7.48-py2.py3-none-any.whl (3.7 MB)
Collecting python-magic==0.4.18
  Downloading python_magic-0.4.18-py2.py3-none-any.whl (8.6 kB)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/frappe/frappe-bench/env/lib/python3.7/site-packages (from boto3==1.4.7->frappe-s3-attachment==0.0.1) (0.10.0)
Collecting s3transfer<0.2.0,>=0.1.10
  Downloading s3transfer-0.1.13-py2.py3-none-any.whl (59 kB)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/frappe/frappe-bench/env/lib/python3.7/site-packages (from botocore==1.7.48->frappe-s3-attachment==0.0.1) (2.8.1)
Requirement already satisfied: docutils>=0.10 in /home/frappe/frappe-bench/env/lib/python3.7/site-packages (from botocore==1.7.48->frappe-s3-attachment==0.0.1) (0.15.2)
Requirement already satisfied: six>=1.5 in /home/frappe/frappe-bench/env/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore==1.7.48->frappe-s3-attachment==0.0.1) (1.14.0)
ERROR: frappe 13.0.0.dev0 has requirement boto3==1.10.18, but you'll have boto3 1.4.7 which is incompatible.
Installing collected packages: botocore, s3transfer, boto3, python-magic, frappe-s3-attachment
  Attempting uninstall: botocore
    Found existing installation: botocore 1.13.50
    Uninstalling botocore-1.13.50:
      Successfully uninstalled botocore-1.13.50
  Attempting uninstall: s3transfer
    Found existing installation: s3transfer 0.2.1
    Uninstalling s3transfer-0.2.1:
      Successfully uninstalled s3transfer-0.2.1
  Attempting uninstall: boto3
    Found existing installation: boto3 1.10.18
    Uninstalling boto3-1.10.18:
      Successfully uninstalled boto3-1.10.18
  Running setup.py develop for frappe-s3-attachment
Successfully installed boto3-1.4.7 botocore-1.7.48 frappe-s3-attachment python-magic-0.4.18 s3transfer-0.1.13
WARNING: You are using pip version 20.1.1; however, version 20.2.4 is available.
You should consider upgrading via the '/home/frappe/frappe-bench/env/bin/python -m pip install --upgrade pip' command.
Removing intermediate container ec1fca2a96b0
 ---> 9af55226beb9
Successfully built 9af55226beb9
Successfully tagged pratikimprowise/frappe-attachments-s3-worker:1.0.1
  1. nginx image docker file. on docker hub build it with docker build --no-cache . -t xxxxxx 
    
FROM bitnami/node:12-prod

COPY ./install_app.sh /install_app RUN /install_app frappe-attachments-s3 https://github.com/zerodhatech/frappe-attachments-s3

FROM frappe/erpnext-nginx:edge

COPY --from=0 /home/frappe/frappe-bench/sites/ /var/www/html/ COPY --from=0 /rsync /rsync RUN echo -n "\nfrappe-attachments-s3" >> /var/www/html/apps.txt

VOLUME [ "/assets" ]

ENTRYPOINT ["/docker-entrypoint.sh"] CMD ["nginx", "-g", "daemon off;"]

docker build logs 
```log
Sending build context to Docker daemon  3.584kB
Step 1/10 : FROM bitnami/node:12-prod
 ---> 946d058b5d18
Step 2/10 : COPY ./install_app.sh /install_app
 ---> f0c7a93365a0
Step 3/10 : RUN /install_app frappe-attachments-s3 https://github.com/zerodhatech/frappe-attachments-s3
 ---> Running in 0b28c633ed54
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  git-man libcurl3-gnutls liberror-perl libgdbm-compat4 libgdbm6 libperl5.28
  libpython2-stdlib libpython2.7-minimal libpython2.7-stdlib mime-support perl
  perl-modules-5.28 python2-minimal python2.7 python2.7-minimal
Suggested packages:
  gettext-base git-daemon-run | git-daemon-sysvinit git-doc git-el git-email
  git-gui gitk gitweb git-cvs git-mediawiki git-svn gdbm-l10n perl-doc
  libterm-readline-gnu-perl | libterm-readline-perl-perl make libb-debug-perl
  liblocale-codes-perl python2-doc python-tk python2.7-doc binutils
  binfmt-support
Recommended packages:
  patch less ssh-client bzip2 file xz-utils netbase
The following NEW packages will be installed:
  git git-man libcurl3-gnutls liberror-perl libgdbm-compat4 libgdbm6
  libperl5.28 libpython2-stdlib libpython2.7-minimal libpython2.7-stdlib
  mime-support perl perl-modules-5.28 python2 python2-minimal python2.7
  python2.7-minimal
0 upgraded, 17 newly installed, 0 to remove and 0 not upgraded.
Need to get 18.8 MB of archives.
After this operation, 102 MB of additional disk space will be used.
Get:1 http://deb.debian.org/debian buster/main amd64 perl-modules-5.28 all 5.28.1-6+deb10u1 [2873 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 libgdbm6 amd64 1.18.1-4 [64.7 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 libgdbm-compat4 amd64 1.18.1-4 [44.1 kB]
Get:4 http://deb.debian.org/debian buster/main amd64 libperl5.28 amd64 5.28.1-6+deb10u1 [3894 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 perl amd64 5.28.1-6+deb10u1 [204 kB]
Get:6 http://deb.debian.org/debian buster/main amd64 libpython2.7-minimal amd64 2.7.16-2+deb10u1 [395 kB]
Get:7 http://deb.debian.org/debian buster/main amd64 python2.7-minimal amd64 2.7.16-2+deb10u1 [1369 kB]
Get:8 http://deb.debian.org/debian buster/main amd64 python2-minimal amd64 2.7.16-1 [41.4 kB]
Get:9 http://deb.debian.org/debian buster/main amd64 mime-support all 3.62 [37.2 kB]
Get:10 http://deb.debian.org/debian buster/main amd64 libpython2.7-stdlib amd64 2.7.16-2+deb10u1 [1912 kB]
Get:11 http://deb.debian.org/debian buster/main amd64 python2.7 amd64 2.7.16-2+deb10u1 [305 kB]
Get:12 http://deb.debian.org/debian buster/main amd64 libpython2-stdlib amd64 2.7.16-1 [20.8 kB]
Get:13 http://deb.debian.org/debian buster/main amd64 python2 amd64 2.7.16-1 [41.6 kB]
Get:14 http://deb.debian.org/debian buster/main amd64 libcurl3-gnutls amd64 7.64.0-4+deb10u1 [330 kB]
Get:15 http://deb.debian.org/debian buster/main amd64 liberror-perl all 0.17027-2 [30.9 kB]
Get:16 http://deb.debian.org/debian buster/main amd64 git-man all 1:2.20.1-2+deb10u3 [1620 kB]
Get:17 http://deb.debian.org/debian buster/main amd64 git amd64 1:2.20.1-2+deb10u3 [5633 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 18.8 MB in 0s (38.3 MB/s)
Selecting previously unselected package perl-modules-5.28.
(Reading database ... 9415 files and directories currently installed.)
Preparing to unpack .../00-perl-modules-5.28_5.28.1-6+deb10u1_all.deb ...
Unpacking perl-modules-5.28 (5.28.1-6+deb10u1) ...
Selecting previously unselected package libgdbm6:amd64.
Preparing to unpack .../01-libgdbm6_1.18.1-4_amd64.deb ...
Unpacking libgdbm6:amd64 (1.18.1-4) ...
Selecting previously unselected package libgdbm-compat4:amd64.
Preparing to unpack .../02-libgdbm-compat4_1.18.1-4_amd64.deb ...
Unpacking libgdbm-compat4:amd64 (1.18.1-4) ...
Selecting previously unselected package libperl5.28:amd64.
Preparing to unpack .../03-libperl5.28_5.28.1-6+deb10u1_amd64.deb ...
Unpacking libperl5.28:amd64 (5.28.1-6+deb10u1) ...
Selecting previously unselected package perl.
Preparing to unpack .../04-perl_5.28.1-6+deb10u1_amd64.deb ...
Unpacking perl (5.28.1-6+deb10u1) ...
Selecting previously unselected package libpython2.7-minimal:amd64.
Preparing to unpack .../05-libpython2.7-minimal_2.7.16-2+deb10u1_amd64.deb ...
Unpacking libpython2.7-minimal:amd64 (2.7.16-2+deb10u1) ...
Selecting previously unselected package python2.7-minimal.
Preparing to unpack .../06-python2.7-minimal_2.7.16-2+deb10u1_amd64.deb ...
Unpacking python2.7-minimal (2.7.16-2+deb10u1) ...
Selecting previously unselected package python2-minimal.
Preparing to unpack .../07-python2-minimal_2.7.16-1_amd64.deb ...
Unpacking python2-minimal (2.7.16-1) ...
Selecting previously unselected package mime-support.
Preparing to unpack .../08-mime-support_3.62_all.deb ...
Unpacking mime-support (3.62) ...
Selecting previously unselected package libpython2.7-stdlib:amd64.
Preparing to unpack .../09-libpython2.7-stdlib_2.7.16-2+deb10u1_amd64.deb ...
Unpacking libpython2.7-stdlib:amd64 (2.7.16-2+deb10u1) ...
Selecting previously unselected package python2.7.
Preparing to unpack .../10-python2.7_2.7.16-2+deb10u1_amd64.deb ...
Unpacking python2.7 (2.7.16-2+deb10u1) ...
Selecting previously unselected package libpython2-stdlib:amd64.
Preparing to unpack .../11-libpython2-stdlib_2.7.16-1_amd64.deb ...
Unpacking libpython2-stdlib:amd64 (2.7.16-1) ...
Setting up libpython2.7-minimal:amd64 (2.7.16-2+deb10u1) ...
Setting up python2.7-minimal (2.7.16-2+deb10u1) ...
Setting up python2-minimal (2.7.16-1) ...
Selecting previously unselected package python2.
(Reading database ... 12135 files and directories currently installed.)
Preparing to unpack .../python2_2.7.16-1_amd64.deb ...
Unpacking python2 (2.7.16-1) ...
Selecting previously unselected package libcurl3-gnutls:amd64.
Preparing to unpack .../libcurl3-gnutls_7.64.0-4+deb10u1_amd64.deb ...
Unpacking libcurl3-gnutls:amd64 (7.64.0-4+deb10u1) ...
Selecting previously unselected package liberror-perl.
Preparing to unpack .../liberror-perl_0.17027-2_all.deb ...
Unpacking liberror-perl (0.17027-2) ...
Selecting previously unselected package git-man.
Preparing to unpack .../git-man_1%3a2.20.1-2+deb10u3_all.deb ...
Unpacking git-man (1:2.20.1-2+deb10u3) ...
Selecting previously unselected package git.
Preparing to unpack .../git_1%3a2.20.1-2+deb10u3_amd64.deb ...
Unpacking git (1:2.20.1-2+deb10u3) ...
Setting up perl-modules-5.28 (5.28.1-6+deb10u1) ...
Setting up mime-support (3.62) ...
Setting up libcurl3-gnutls:amd64 (7.64.0-4+deb10u1) ...
Setting up libpython2.7-stdlib:amd64 (2.7.16-2+deb10u1) ...
Setting up git-man (1:2.20.1-2+deb10u3) ...
Setting up libgdbm6:amd64 (1.18.1-4) ...
Setting up python2.7 (2.7.16-2+deb10u1) ...
Setting up libpython2-stdlib:amd64 (2.7.16-1) ...
Setting up python2 (2.7.16-1) ...
Setting up libgdbm-compat4:amd64 (1.18.1-4) ...
Setting up libperl5.28:amd64 (5.28.1-6+deb10u1) ...
Setting up perl (5.28.1-6+deb10u1) ...
Setting up liberror-perl (0.17027-2) ...
Setting up git (1:2.20.1-2+deb10u3) ...
Processing triggers for libc-bin (2.28-10) ...
Cloning into 'frappe'...
Cloning into 'frappe-attachments-s3'...
yarn install v1.22.10
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
warning " > bootstrap@4.4.1" has unmet peer dependency "jquery@1.9.1 - 3".
warning " > bootstrap@4.4.1" has unmet peer dependency "popper.js@^1.16.0".
[4/4] Building fresh packages...
$ yarn run snyk-protect
yarn run v1.22.10
$ snyk protect
Successfully applied Snyk patches

Done in 10.50s.
Done in 79.96s.
yarn run v1.22.10
$ FRAPPE_ENV=production node rollup/build.js --app frappe-attachments-s3
Production mode
✔ Built js/moment-bundle.min.js
✔ Built js/libs.min.js
Done in 1.58s.
yarn install v1.22.10
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
warning " > bootstrap@4.4.1" has unmet peer dependency "jquery@1.9.1 - 3".
warning " > bootstrap@4.4.1" has unmet peer dependency "popper.js@^1.16.0".
[4/4] Building fresh packages...
Done in 23.49s.
yarn add v1.22.10
[1/4] Resolving packages...
warning node-sass > node-gyp > request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
[2/4] Fetching packages...
[3/4] Linking dependencies...
warning " > bootstrap@4.4.1" has unmet peer dependency "jquery@1.9.1 - 3".
warning " > bootstrap@4.4.1" has unmet peer dependency "popper.js@^1.16.0".
[4/4] Building fresh packages...
success Saved lockfile.
success Saved 10 new dependencies.
info Direct dependencies
└─ node-sass@5.0.0
info All dependencies
├─ chownr@2.0.0
├─ cross-spawn@7.0.3
├─ env-paths@2.2.0
├─ fs-minipass@2.1.0
├─ minizlib@2.1.2
├─ node-gyp@7.1.2
├─ node-sass@5.0.0
├─ shebang-command@2.0.0
├─ shebang-regex@3.0.0
└─ tar@6.0.5
$ yarn run snyk-protect
yarn run v1.22.10
$ snyk protect
Successfully applied Snyk patches

Done in 13.01s.
Done in 38.65s.
cp: cannot stat '/home/frappe/frappe-bench/apps/frappe-attachments-s3/frappe-attachments-s3/public/*': No such file or directory
Removing intermediate container 0b28c633ed54
 ---> 30d81b69027f
Step 4/10 : FROM frappe/erpnext-nginx:edge
 ---> dd4835b314f1
Step 5/10 : COPY --from=0 /home/frappe/frappe-bench/sites/ /var/www/html/
 ---> 2dc9ed209b3f
Step 6/10 : COPY --from=0 /rsync /rsync
 ---> ab0f4270f9d8
Step 7/10 : RUN echo -n "\nfrappe-attachments-s3" >> /var/www/html/apps.txt
 ---> Running in cfa197bd7125
Removing intermediate container cfa197bd7125
 ---> 2de4459592e9
Step 8/10 : VOLUME [ "/assets" ]
 ---> Running in 7b0a46c5ecc9
Removing intermediate container 7b0a46c5ecc9
 ---> f269a997b1cf
Step 9/10 : ENTRYPOINT ["/docker-entrypoint.sh"]
 ---> Running in a2d1de8dd4ce
Removing intermediate container a2d1de8dd4ce
 ---> 1beecf6487cd
Step 10/10 : CMD ["nginx", "-g", "daemon off;"]
 ---> Running in 673b1a721d19
Removing intermediate container 673b1a721d19
 ---> 1025638c5aff
Successfully built 1025638c5aff
Successfully tagged pratikimprowise/frappe-attachments-s3-nginx:1.0.1
  1. added these to values.yaml of erpnext helm chart as per these Manual way step, 
    
nginxImage:
repository: docker.io/pratikimprowise/frappe-attachments-s3-nginx
tag: 1.0.1
pullPolicy: Always

pythonImage: repository: docker.io/pratikimprowise/frappe-attachments-s3-worker tag: 1.0.1 pullPolicy: Always

execute it 
```sh
helm install frappe . --create-namespace --namespace erpnext  -f values.yaml

image

Version

Docker v19.06 k3s version v1.19.3 helm version v3.3.4 frappe-attachments-s3-worker frappe-attachments-s3-nginx

pratikbin commented 3 years ago

figured. fork this repo with name frappe_attachments_s3 and rest as per erpnext steps