Job for uwsgi.service failed

[valeriodeluca]

Hi eb, I'm here, Ubuntu 20.04 Core installation Install and configure NGINX https://docs.geonode.org/en/master/install/advanced/core/#install-and-configure-nginx

When I restart the uwsgi service: sudo service uwsgi restart

I get: Job for uwsgi.service failed because the control process exited with error code. See "systemctl status uwsgi.service" and "journalctl -xe" for details.

The service status is: ● uwsgi.service - LSB: Start/stop uWSGI server instance(s) Loaded: loaded (/etc/init.d/uwsgi; generated) Active: failed (Result: exit-code) since Sun 2021-01-03 20:00:21 CET; 2min 51s ago Docs: man:systemd-sysv-generator(8) Process: 25449 ExecStart=/etc/init.d/uwsgi start (code=exited, status=1/FAILURE)

Thanks

[jlsevillano]

Hi, I followed / following the same doc and I have exactly the same issue:

● uwsgi.service - LSB: Start/stop uWSGI server instance(s) Loaded: loaded (/etc/init.d/uwsgi; generated) Active: failed (Result: exit-code) since Fri 2021-01-08 08:33:11 UTC; 12s ago Docs: man:systemd-sysv-generator(8) Process: 53545 ExecStart=/etc/init.d/uwsgi start (code=exited, status=1/FAILURE)

Jan 08 08:33:08 geonode systemd[1]: Starting LSB: Start/stop uWSGI server instance(s)... Jan 08 08:33:08 geonode uwsgi[53545]: * Starting app server(s) uwsgi Jan 08 08:33:11 geonode uwsgi[53545]: ...fail! Jan 08 08:33:11 geonode systemd[1]: uwsgi.service: Control process exited, code=exited, status=1/FAILURE Jan 08 08:33:11 geonode systemd[1]: uwsgi.service: Failed with result 'exit-code'. Jan 08 08:33:11 geonode systemd[1]: Failed to start LSB: Start/stop uWSGI server instance(s).

[jlsevillano]

More info: Logs says: ModuleNotFoundError: No module named 'geonode.local_settings' Fri Jan 8 09:12:36 2021 - unable to load app 0 (mountpoint='') (callable not found or import error) Fri Jan 8 09:12:36 2021 - no app loaded. GAME OVER Fri Jan 8 09:12:36 2021 - VACUUM: unix socket /run/uwsgi/app/geonode/socket removed.

[valeriodeluca]

@afabiani @gannebamm @t-book any suggestions? Thanks very much

[t-book]

No module named 'geonode.local_settings'

Try to run pip install -e . inside your geonode folder with ativated env.

https://github.com/GeoNode/geonode/issues/6050

[jlsevillano]

I tried. pip install -e .

Result: Installing collected packages: GeoNode Running setup.py develop for GeoNode Successfully installed GeoNode WARNING: You are using pip version 20.2.4; however, version 20.3.3 is available. You should consider upgrading via the '/home/geonode/.virtualenvs/geonode/bin/python -m pip install --upgrade pip' command. And: (geonode) geonode@geonode:/opt/geonode$ sudo service uwsgi restart

And same error :(

Second try: pip install -e . --upgrade

Result: Installing collected packages: GeoNode Attempting uninstall: GeoNode Found existing installation: GeoNode 3.1.0 Uninstalling GeoNode-3.1.0: Successfully uninstalled GeoNode-3.1.0 Running setup.py develop for GeoNode Successfully installed GeoNode WARNING: You are using pip version 20.2.4; however, version 20.3.3 is available. You should consider upgrading via the '/home/geonode/.virtualenvs/geonode/bin/python -m pip install --upgrade pip' command. (geonode) geonode@geonode:/opt/geonode$ sudo service uwsgi restart Job for uwsgi.service failed because the control process exited with error code. See "systemctl status uwsgi.service" and "journalctl -xe" for details. (geonode) geonode@geonode:/opt/geonode$

[afabiani]

That usually depends by the default UWSGI service being not able to load the virtualenv.

I'll send you some troubleshooting lines along with an alternative solution in a few. It will take some time to collect all the stuff and write down a comprehensive guideline.

It would be probably good then to update the documentation as well.

[t-book]

Also, a good startingpoint to get more info might be /opt/data/logs/geonode.log

[jlsevillano]

@afabiani @t-book Thanks a lot, asap I'll send you /opt/data/logs/geonode.log

[valeriodeluca]

Same @jlsevillano issues for me. @afabiani thanks for your troubleshooting lines. Can I help with the documetation update?

[afabiani]

Throubleshooting the UWSGI config

First of all we need to understand if there's some issue with the uwsgi.ini syntax itself.

A quick way to test it, it is to run it directly.

1. Make sure no uwsgi service is running. If so, forcibly kill all the running processes.

2. Make a try, both in and out the python virtualenv by running the following command:

   uwsgi -c [/full_path_to/uwsgi].ini

   If no errors occurs, you should be able to see something similar to the output below:

   geosolutions@ns300380:~/geonode$ uwsgi -c /etc/uwsgi/apps-enabled/geonode.ini
   [uWSGI] getting INI configuration from /etc/uwsgi/apps-enabled/geonode.ini
   *** Starting uWSGI 2.0.19.1 (64bit) on [Sat Jan  9 16:50:27 2021] ***
   compiled with version: 7.5.0 on 14 September 2020 11:11:21
   os: Linux-4.15.0-111-generic #112-Ubuntu SMP Thu Jul 9 20:32:34 UTC 2020
   nodename: ns300380
   machine: x86_64
   clock source: unix
   pcre jit disabled
   detected number of CPU cores: 8
   current working directory: /home/geosolutions/geonode
   detected binary path: /usr/local/src/uwsgi-2.0.19.1/uwsgi
   chdir() to /home/geosolutions/geonode/
   your processes number limit is 63873
   your memory page size is 4096 bytes
    *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers ***
    detected max file descriptor number: 1024
    lock engine: pthread robust mutexes
    thunder lock: disabled (you can enable it with --thunder-lock)
    [busyness] settings: min=20%, max=70%, overload=1, multiplier=30, respawn penalty=2
    [busyness] backlog alert is set to 16 request(s), step is 2
    [busyness] backlog non-zero alert is set to 60 second(s)
    uwsgi socket 0 bound to TCP address 0.0.0.0:8000 fd 3
    Python version: 3.8.7 (default, Dec 21 2020, 20:10:35)  [GCC 7.5.0]
    Set PythonHome to /home/geosolutions/Envs/geonode3
    Python main interpreter initialized at 0x55de0dbc53d0
    python threads support enabled
    your server socket listen backlog is limited to 100 connections
    your mercy for graceful operations on workers is 600 seconds
    mapped 13105368 bytes (12798 KB) for 128 cores
    *** Operational MODE: preforking ***
    [Errno 2] No such file or directory: 'spatialite'
    WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x55de0dbc53d0 pid: 24567 (default app)
    mountpoint  already configured. skip.
    spawned uWSGI master process (pid: 24567)
    spawned uWSGI worker 1 (pid: 24583, cores: 1)
    spawned uWSGI worker 2 (pid: 24584, cores: 1)
    spawned uWSGI worker 3 (pid: 24585, cores: 1)
    spawned uWSGI worker 4 (pid: 24586, cores: 1)
    spawned uWSGI worker 5 (pid: 24587, cores: 1)
    spawned uWSGI worker 6 (pid: 24588, cores: 1)
    spawned uWSGI worker 7 (pid: 24589, cores: 1)
    spawned uWSGI worker 8 (pid: 24590, cores: 1)
    spawned uWSGI worker 9 (pid: 24591, cores: 1)
    spawned uWSGI worker 10 (pid: 24592, cores: 1)
    spawned uWSGI worker 11 (pid: 24593, cores: 1)
    spawned uWSGI worker 12 (pid: 24594, cores: 1)
    spawned uWSGI worker 13 (pid: 24595, cores: 1)
    spawned uWSGI worker 14 (pid: 24596, cores: 1)
    spawned uWSGI worker 15 (pid: 24597, cores: 1)
    spawned uWSGI worker 16 (pid: 24598, cores: 1)

Test it with NGINX too. Change the uwsgi-sockets accordingly, if needed.

Creating a custom UWSGI service [Ubuntu]

Whenever you make the previous configuration working correctly, and the standard UWSGI service still does not work, you might need to create a custom service yourself.

In some configurations, I personally experienced the default UWSGI service not being able to recognize the Python virtual environment somehow, and therefore not bwing able to start it by using the correct interpreter.

In a similar case, what I usually do to fix the issue is:

1. Create a Service

   sudo vim /etc/systemd/system/geonode-uwsgi.service

   [Unit]
   Description=GeoNode UWSGI Service
   
   [Service]
   User=geosolutions
   # The configuration file application.properties should be here:
   
   #change this to your workspace
   WorkingDirectory=/home/geosolutions/geonode
   
   #path to executable.
   #executable is a bash script which calls jar file
   ExecStart=/home/geosolutions/geonode-uwsgi.sh
   
   SuccessExitStatus=143
   TimeoutStopSec=10
   Restart=on-failure
   RestartSec=5
   
   [Install]
   WantedBy=multi-user.target

2. Create a Bash Script to Call Your Service

   vim /home/geosolutions/geonode-uwsgi.sh

   #!/bin/bash
   source ~/Envs/geonode3/bin/activate
   uwsgi -c /etc/uwsgi/apps-enabled/geonode.ini

   chmod +x /home/geosolutions/geonode-uwsgi.sh

3. Start the Service

   sudo systemctl daemon-reload
   sudo systemctl start geonode-uwsgi
   sudo systemctl status geonode-uwsgi

   If everything is working, don't forget to enable the new service and disable the default one

   sudo systemctl disable uwsgi
   sudo systemctl enable geonode-uwsgi

[valeriodeluca]

@afabiani thanks very much, I try it right away.

[gannebamm]

I encountered the same issue and trying the debugging @afabiani described above. Some questions:

should uwsgi be called as sudo?

(geonode) tisdex@zi-sws-004v-dmz:/opt/geonode$ uwsgi -c /etc/uwsgi/apps-available/geonode.ini
[uWSGI] getting INI configuration from /etc/uwsgi/apps-available/geonode.ini
open("./python3_plugin.so"): No such file or directory [core/utils.c line 3732]
!!! UNABLE to load uWSGI plugin: ./python3_plugin.so: cannot open shared object file: No such file or directory !!!
(geonode) tisdex@zi-sws-004v-dmz:/opt/geonode$ open("/opt/data/logs/geonode.log"): Permission denied [core/logging.c line 288]

If I do so it only displays:

(geonode) tisdex@zi-sws-004v-dmz:/opt/geonode$ sudo uwsgi -c /etc/uwsgi/apps-enabled/geonode.ini
[uWSGI] getting INI configuration from /etc/uwsgi/apps-enabled/geonode.ini
(geonode) tisdex@zi-sws-004v-dmz:/opt/geonode$

The workers will not get spawned. So the .ini file is somehow wrong?

[gannebamm]

After taking a look at the /opt/data/logs/geonode.log

I do see the output which states my usage of sudo uwsgi as dangerous - which it is and won´t be after finishing the debugging:

*** Starting uWSGI 2.0.18-debian (64bit) on [Tue Jan 12 11:16:04 2021] ***
compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55
os: Linux-5.4.0-60-generic #67-Ubuntu SMP Tue Jan 5 18:31:36 UTC 2021
nodename: zi-sws-004v-dmz
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 24
current working directory: /opt/geonode
writing pidfile to /tmp/geonode.pid
detected binary path: /usr/bin/uwsgi-core
uWSGI running as root, you can use --uid/--gid/--chroot options
setgid() to 33
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
chdir() to /opt/geonode
your processes number limit is 31392
your memory page size is 4096 bytes
 *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers ***
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 0.0.0.0:8000 fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
setgid() to 33
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Python version: 3.8.5 (default, Jul 28 2020, 12:59:40)  [GCC 9.3.0]
PEP 405 virtualenv detected: /home/tisdex/.virtualenvs/geonode
Set PythonHome to /home/tisdex/.virtualenvs/geonode
Python main interpreter initialized at 0x55a163b01940
uWSGI running as root, you can use --uid/--gid/--chroot options
setgid() to 33
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 13105368 bytes (12798 KB) for 128 cores
*** Operational MODE: preforking ***
WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x55a163b01940 pid: 3121 (default app)
uWSGI running as root, you can use --uid/--gid/--chroot options
setgid() to 33
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
spawned uWSGI master process (pid: 3121)
spawned uWSGI worker 1 (pid: 3153, cores: 1)
spawned uWSGI worker 2 (pid: 3154, cores: 1)
spawned uWSGI worker 3 (pid: 3155, cores: 1)
spawned uWSGI worker 4 (pid: 3156, cores: 1)
spawned uWSGI worker 5 (pid: 3157, cores: 1)
spawned uWSGI worker 6 (pid: 3158, cores: 1)
spawned uWSGI worker 7 (pid: 3159, cores: 1)
spawned uWSGI worker 8 (pid: 3160, cores: 1)
spawned uWSGI worker 9 (pid: 3161, cores: 1)
spawned uWSGI worker 10 (pid: 3162, cores: 1)
spawned uWSGI worker 11 (pid: 3163, cores: 1)
spawned uWSGI worker 12 (pid: 3164, cores: 1)
spawned uWSGI worker 13 (pid: 3165, cores: 1)
spawned uWSGI worker 14 (pid: 3166, cores: 1)
spawned uWSGI worker 15 (pid: 3167, cores: 1)
spawned uWSGI worker 16 (pid: 3168, cores: 1)
unable to find requested cheaper algorithm, falling back to spare
worker 1 killed successfully (pid: 3153)
uWSGI worker 1 cheaped.
worker 2 killed successfully (pid: 3154)
uWSGI worker 2 cheaped.
worker 3 killed successfully (pid: 3155)
uWSGI worker 3 cheaped.
worker 4 killed successfully (pid: 3156)
uWSGI worker 4 cheaped.
worker 5 killed successfully (pid: 3157)
uWSGI worker 5 cheaped.
worker 6 killed successfully (pid: 3158)
uWSGI worker 6 cheaped.
worker 7 killed successfully (pid: 3159)
uWSGI worker 7 cheaped.
worker 8 killed successfully (pid: 3160)
uWSGI worker 8 cheaped.

So this seems ok? It does load the virtual env properly.

[afabiani]

@gannebamm yeps, that is ok

[afabiani]

@gannebamm by the way, if you want to "daemonize" the process, just add the line

daemonize = path_to_log_file

To your uwsgi.ini

[gannebamm]

I tried the new service definition solution but now the new geonode-uwsgi.service does fail with:

Jan 12 11:28:48 zi-sws-004v-dmz geonode-uwsgi.sh[3567]: !!! UNABLE to load uWSGI plugin:  ./python3_plugin.so: cannot open shared object file: No such file or directory !!!

[afabiani]

@gannebamm just remove the line

plugin

It will use the venv built one.

[gannebamm]

Thanks @afabiani ! We are making progress but are not there, yet. For some reason the uwsgi service is in a restart loop:

geonode.log:

*** Starting uWSGI 2.0.19.1 (64bit) on [Tue Jan 12 11:53:37 2021] ***
compiled with version: 9.3.0 on 11 January 2021 15:48:46
os: Linux-5.4.0-60-generic #67-Ubuntu SMP Tue Jan 5 18:31:36 UTC 2021
nodename: zi-sws-004v-dmz
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 24
current working directory: /opt/geonode
writing pidfile to /tmp/geonode.pid
detected binary path: /home/tisdex/.virtualenvs/geonode/bin/uwsgi
chdir() to /opt/geonode
your processes number limit is 31392
your memory page size is 4096 bytes
 *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers ***
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
[busyness] settings: min=20%, max=70%, overload=1, multiplier=30, respawn penalty=2
[busyness] backlog alert is set to 16 request(s), step is 2
[busyness] backlog non-zero alert is set to 60 second(s)
uwsgi socket 0 bound to TCP address 0.0.0.0:8000 fd 3
Python version: 3.8.5 (default, Jul 28 2020, 12:59:40)  [GCC 9.3.0]
PEP 405 virtualenv detected: /home/tisdex/.virtualenvs/geonode
Set PythonHome to /home/tisdex/.virtualenvs/geonode
Python main interpreter initialized at 0x55ef62204530
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 13105368 bytes (12798 KB) for 128 cores
*** Operational MODE: preforking ***
WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x55ef62204530 pid: 4710 (default app)
spawned uWSGI master process (pid: 4710)
spawned uWSGI worker 1 (pid: 4742, cores: 1)
spawned uWSGI worker 2 (pid: 4743, cores: 1)
spawned uWSGI worker 3 (pid: 4744, cores: 1)
spawned uWSGI worker 4 (pid: 4745, cores: 1)
spawned uWSGI worker 5 (pid: 4746, cores: 1)
spawned uWSGI worker 6 (pid: 4747, cores: 1)
spawned uWSGI worker 7 (pid: 4748, cores: 1)
spawned uWSGI worker 8 (pid: 4749, cores: 1)
spawned uWSGI worker 9 (pid: 4750, cores: 1)
spawned uWSGI worker 10 (pid: 4751, cores: 1)
spawned uWSGI worker 11 (pid: 4752, cores: 1)
spawned uWSGI worker 12 (pid: 4753, cores: 1)
spawned uWSGI worker 13 (pid: 4754, cores: 1)
spawned uWSGI worker 14 (pid: 4755, cores: 1)
spawned uWSGI worker 15 (pid: 4756, cores: 1)
spawned uWSGI worker 16 (pid: 4757, cores: 1)
*** Starting uWSGI 2.0.19.1 (64bit) on [Tue Jan 12 11:53:52 2021] ***
compiled with version: 9.3.0 on 11 January 2021 15:48:46
os: Linux-5.4.0-60-generic #67-Ubuntu SMP Tue Jan 5 18:31:36 UTC 2021
nodename: zi-sws-004v-dmz
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 24
current working directory: /opt/geonode
writing pidfile to /tmp/geonode.pid
detected binary path: /home/tisdex/.virtualenvs/geonode/bin/uwsgi
chdir() to /opt/geonode
your processes number limit is 31392
your memory page size is 4096 bytes
 *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers ***
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
[busyness] settings: min=20%, max=70%, overload=1, multiplier=30, respawn penalty=2
[busyness] backlog alert is set to 16 request(s), step is 2
[busyness] backlog non-zero alert is set to 60 second(s)
uwsgi socket 0 bound to TCP address 0.0.0.0:8000 fd 3
Python version: 3.8.5 (default, Jul 28 2020, 12:59:40)  [GCC 9.3.0]
PEP 405 virtualenv detected: /home/tisdex/.virtualenvs/geonode
Set PythonHome to /home/tisdex/.virtualenvs/geonode
Python main interpreter initialized at 0x55ae6c681530
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 13105368 bytes (12798 KB) for 128 cores
*** Operational MODE: preforking ***
WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x55ae6c681530 pid: 4783 (default app)
spawned uWSGI master process (pid: 4783)
spawned uWSGI worker 1 (pid: 4815, cores: 1)
spawned uWSGI worker 2 (pid: 4816, cores: 1)
spawned uWSGI worker 3 (pid: 4817, cores: 1)
spawned uWSGI worker 4 (pid: 4818, cores: 1)
spawned uWSGI worker 5 (pid: 4819, cores: 1)
spawned uWSGI worker 6 (pid: 4820, cores: 1)
spawned uWSGI worker 7 (pid: 4821, cores: 1)
spawned uWSGI worker 8 (pid: 4822, cores: 1)
spawned uWSGI worker 9 (pid: 4823, cores: 1)
spawned uWSGI worker 10 (pid: 4824, cores: 1)
spawned uWSGI worker 11 (pid: 4825, cores: 1)
spawned uWSGI worker 12 (pid: 4826, cores: 1)
spawned uWSGI worker 13 (pid: 4827, cores: 1)
spawned uWSGI worker 14 (pid: 4828, cores: 1)
spawned uWSGI worker 15 (pid: 4829, cores: 1)
spawned uWSGI worker 16 (pid: 4830, cores: 1)

why will it start again?

checking the services status:

(geonode) tisdex@zi-sws-004v-dmz:/opt/data/logs$ sudo service geonode-uwsgi status
● geonode-uwsgi.service - GeoNode UWSGI Service
     Loaded: loaded (/etc/systemd/system/geonode-uwsgi.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: timeout) since Tue 2021-01-12 11:54:03 UTC; 4s ago
    Process: 4769 ExecStart=/opt/geonode/geonode-uwsgi.sh (code=exited, status=0/SUCCESS)
   Main PID: 4769 (code=exited, status=0/SUCCESS)

[afabiani]

Uhm, that's strange... make sure you don't have other services trying to restart uwsgi somehow and also cleanup all pending uwsgi processes before starting the service

sudo pkill -9 -f uwsgi

[gannebamm]

It is still in a reboot loop, if I use the service unit. I have killed everything uwsgi related and disabled the default uwsgi service. Then there was an issue with the geonode.pid file still trailing and I tried to change it to use a geonode_new.pid file. I dumped the service unit file and gone back to the manual uwsgi command.

The result of using sudo uwsgi -c /etc/uwsgi/apps-enabled/geonode.ini is now:

*** Operational MODE: preforking ***
WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x560038477940 pid: 30490 (default app)
uWSGI running as root, you can use --uid/--gid/--chroot options
setgid() to 33
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
spawned uWSGI master process (pid: 30490)
spawned uWSGI worker 1 (pid: 30523, cores: 1)
spawned uWSGI worker 2 (pid: 30524, cores: 1)
spawned uWSGI worker 3 (pid: 30525, cores: 1)
spawned uWSGI worker 4 (pid: 30526, cores: 1)
spawned uWSGI worker 5 (pid: 30527, cores: 1)
spawned uWSGI worker 6 (pid: 30528, cores: 1)
spawned uWSGI worker 7 (pid: 30529, cores: 1)
spawned uWSGI worker 8 (pid: 30530, cores: 1)
spawned uWSGI worker 9 (pid: 30531, cores: 1)
spawned uWSGI worker 10 (pid: 30532, cores: 1)
spawned uWSGI worker 11 (pid: 30533, cores: 1)
spawned uWSGI worker 12 (pid: 30534, cores: 1)
spawned uWSGI worker 13 (pid: 30535, cores: 1)
spawned uWSGI worker 14 (pid: 30536, cores: 1)
spawned uWSGI worker 15 (pid: 30537, cores: 1)
spawned uWSGI worker 16 (pid: 30538, cores: 1)
unable to find requested cheaper algorithm, falling back to spare
worker 1 killed successfully (pid: 30523)
uWSGI worker 1 cheaped.
worker 2 killed successfully (pid: 30524)
uWSGI worker 2 cheaped.
worker 3 killed successfully (pid: 30525)
uWSGI worker 3 cheaped.
worker 4 killed successfully (pid: 30526)
uWSGI worker 4 cheaped.
worker 5 killed successfully (pid: 30527)
uWSGI worker 5 cheaped.
worker 6 killed successfully (pid: 30528)
uWSGI worker 6 cheaped.
worker 7 killed successfully (pid: 30529)
uWSGI worker 7 cheaped.
worker 8 killed successfully (pid: 30530)
uWSGI worker 8 cheaped.

For some reason, the workers will get killed, which they should not?

But they are not in a reboot loop it seems.

[afabiani]

@gannebamm if you mean those killed successfully statements, it is normal. That is the cheaper algorythm respawning the unused workers.

[gannebamm]

ok, using the manual uwsgi start it does seem to work. But the unit file does not:

(geonode) tisdex@zi-sws-004v-dmz:/opt/geonode$ sudo systemctl start geonode-uwsgi
(geonode) tisdex@zi-sws-004v-dmz:/opt/geonode$ sudo systemctl status geonode-uwsgi
● geonode-uwsgi.service - GeoNode UWSGI Service
     Loaded: loaded (/etc/systemd/system/geonode-uwsgi.service; disabled; vendor preset: enabled)
     Active: failed (Result: timeout) since Tue 2021-01-12 14:13:49 UTC; 1min 44s ago
    Process: 31320 ExecStart=/opt/geonode/geonode-uwsgi.sh (code=exited, status=0/SUCCESS)
   Main PID: 31320 (code=exited, status=0/SUCCESS)

Jan 12 14:13:49 zi-sws-004v-dmz systemd[1]: geonode-uwsgi.service: Killing process 31380 (uwsgi) with signal SIGKILL.
Jan 12 14:13:49 zi-sws-004v-dmz systemd[1]: geonode-uwsgi.service: Killing process 31381 (uwsgi) with signal SIGKILL.
Jan 12 14:13:49 zi-sws-004v-dmz systemd[1]: geonode-uwsgi.service: Killing process 31382 (uwsgi) with signal SIGKILL.
Jan 12 14:13:49 zi-sws-004v-dmz systemd[1]: geonode-uwsgi.service: Killing process 31383 (uwsgi) with signal SIGKILL.
Jan 12 14:13:49 zi-sws-004v-dmz systemd[1]: geonode-uwsgi.service: Killing process 31384 (uwsgi) with signal SIGKILL.
Jan 12 14:13:49 zi-sws-004v-dmz systemd[1]: geonode-uwsgi.service: Killing process 31385 (uwsgi) with signal SIGKILL.
Jan 12 14:13:49 zi-sws-004v-dmz systemd[1]: geonode-uwsgi.service: Killing process 31386 (uwsgi) with signal SIGKILL.
Jan 12 14:13:49 zi-sws-004v-dmz systemd[1]: geonode-uwsgi.service: Killing process 31387 (uwsgi) with signal SIGKILL.
Jan 12 14:13:49 zi-sws-004v-dmz systemd[1]: geonode-uwsgi.service: Killing process 31388 (uwsgi) with signal SIGKILL.
Jan 12 14:13:49 zi-sws-004v-dmz systemd[1]: geonode-uwsgi.service: Failed with result 'timeout'.

It will get killed for some reason. The logfile does not show the issue and does look appropriate (unit file is set to restart=on-success): tisdex@zi-sws-004v-dmz:/home/hoedt_adm$ tail /opt/data/logs/geonode.log -f

*** Starting uWSGI 2.0.19.1 (64bit) on [Tue Jan 12 14:13:39 2021] ***
compiled with version: 9.3.0 on 11 January 2021 15:48:46
os: Linux-5.4.0-60-generic #67-Ubuntu SMP Tue Jan 5 18:31:36 UTC 2021
nodename: zi-sws-004v-dmz
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 24
current working directory: /opt/geonode
writing pidfile to /tmp/geonode_new.pid
detected binary path: /home/tisdex/.virtualenvs/geonode/bin/uwsgi
chdir() to /opt/geonode
your processes number limit is 31392
your memory page size is 4096 bytes
 *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers ***
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
[busyness] settings: min=20%, max=70%, overload=1, multiplier=30, respawn penalty=2
[busyness] backlog alert is set to 16 request(s), step is 2
[busyness] backlog non-zero alert is set to 60 second(s)
uwsgi socket 0 bound to TCP address 0.0.0.0:8000 fd 3
Python version: 3.8.5 (default, Jul 28 2020, 12:59:40)  [GCC 9.3.0]
PEP 405 virtualenv detected: /home/tisdex/.virtualenvs/geonode
Set PythonHome to /home/tisdex/.virtualenvs/geonode
Python main interpreter initialized at 0x5611c2603790
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 13105368 bytes (12798 KB) for 128 cores
*** Operational MODE: preforking ***
WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x5611c2603790 pid: 31326 (default app)
spawned uWSGI master process (pid: 31326)
spawned uWSGI worker 1 (pid: 31373, cores: 1)
spawned uWSGI worker 2 (pid: 31374, cores: 1)
spawned uWSGI worker 3 (pid: 31375, cores: 1)
spawned uWSGI worker 4 (pid: 31376, cores: 1)
spawned uWSGI worker 5 (pid: 31377, cores: 1)
spawned uWSGI worker 6 (pid: 31378, cores: 1)
spawned uWSGI worker 7 (pid: 31379, cores: 1)
spawned uWSGI worker 8 (pid: 31380, cores: 1)
spawned uWSGI worker 9 (pid: 31381, cores: 1)
spawned uWSGI worker 10 (pid: 31382, cores: 1)
spawned uWSGI worker 11 (pid: 31383, cores: 1)
spawned uWSGI worker 12 (pid: 31384, cores: 1)
spawned uWSGI worker 13 (pid: 31385, cores: 1)
spawned uWSGI worker 14 (pid: 31386, cores: 1)
spawned uWSGI worker 15 (pid: 31387, cores: 1)
spawned uWSGI worker 16 (pid: 31388, cores: 1)

The corresponding unit file:

[Unit]
Description=GeoNode UWSGI Service

[Service]
User=tisdex
# The configuration file application.properties should be here:

#change this to your workspace
WorkingDirectory=/opt/geonode

#path to executable.
#executable is a bash script which calls jar file
ExecStart=/opt/geonode/geonode-uwsgi.sh

SuccessExitStatus=143
TimeoutStopSec=10
Restart=on-success #on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

I do not get whats wrong here...

[jlsevillano]

edited by gannebamm for better formatting

Thanks @afabiani ! We are making progress but are not there, yet. For some reason the uwsgi service is not running: Before to execute:

1. Make sure no uwsgi service is running. If so, forcibly kill all the running processes. (we did it sudo pkill -9 -f uwsgi to be sure) 2.- (geonode) geonode@geonode:/opt/geonode$ sudo uwsgi -c /etc/uwsgi/apps-enabled/geonode.ini [uWSGI] getting INI configuration from /etc/uwsgi/apps-enabled/geonode.ini

geonode.log:

*** Starting uWSGI 2.0.18-debian (64bit) on [Tue Jan 12 14:39:38 2021] ***
compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55
os: Linux-5.4.0-51-generic #56-Ubuntu SMP Mon Oct 5 14:28:49 UTC 2020
nodename: geonode
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 2
current working directory: /opt/geonode
writing pidfile to /tmp/geonode.pid
detected binary path: /usr/bin/uwsgi-core
uWSGI running as root, you can use --uid/--gid/--chroot options
setgid() to 33
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
chdir() to /opt/geonode
your processes number limit is 15610
your memory page size is 4096 bytes
 *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers *** 
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 0.0.0.0:8000 fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
setgid() to 33
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
Python version: 3.8.5 (default, Jul 28 2020, 12:59:40)  [GCC 9.3.0]
PEP 405 virtualenv detected: /home/geonode/.virtualenvs/geonode
Set PythonHome to /home/geonode/.virtualenvs/geonode
Python main interpreter initialized at 0x55c2f951f940
uWSGI running as root, you can use --uid/--gid/--chroot options
setgid() to 33
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 13105368 bytes (12798 KB) for 128 cores
*** Operational MODE: preforking ***
Traceback (most recent call last):
  File "./geonode/wsgi.py", line 30, in <module>
    application = get_wsgi_application()
  File "/home/geonode/.virtualenvs/geonode/lib/python3.8/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
    django.setup(set_prefix=False)
  File "/home/geonode/.virtualenvs/geonode/lib/python3.8/site-packages/django/__init__.py", line 19, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
  File "/home/geonode/.virtualenvs/geonode/lib/python3.8/site-packages/django/conf/__init__.py", line 79, in __getattr__
    self._setup(name)
  File "/home/geonode/.virtualenvs/geonode/lib/python3.8/site-packages/django/conf/__init__.py", line 66, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/geonode/.virtualenvs/geonode/lib/python3.8/site-packages/django/conf/__init__.py", line 157, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'geonode.local_settings'
unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. GAME OVER ***
VACUUM: pidfile removed.

And is impossible to run:

(geonode) geonode@geonode:/opt/geonode$ sudo service uwsgi restart
Job for uwsgi.service failed because the control process exited with error code.
See "systemctl status uwsgi.service" and "journalctl -xe" for details.
(geonode) geonode@geonode:/opt/geonode$

(geonode) geonode@geonode:/opt/geonode$ sudo journalctl -xe
-- Subject: A start job for unit uwsgi.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit uwsgi.service has begun execution.
--
-- The job identifier is 3402.
Jan 12 14:45:24 geonode uwsgi[52152]:  * Starting app server(s) uwsgi
Jan 12 14:45:27 geonode uwsgi[52152]:    ...fail!
Jan 12 14:45:27 geonode systemd[1]: uwsgi.service: Control process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit uwsgi.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Jan 12 14:45:27 geonode systemd[1]: uwsgi.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit uwsgi.service has entered the 'failed' state with result 'exit-code'.
Jan 12 14:45:27 geonode systemd[1]: Failed to start LSB: Start/stop uWSGI server instance(s).
-- Subject: A start job for unit uwsgi.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit uwsgi.service has finished with a failure.
--
-- The job identifier is 3402 and the job result is failed.
Jan 12 14:45:27 geonode sudo[52146]: pam_unix(sudo:session): session closed for user root
Jan 12 14:45:39 geonode sshd[52394]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=139.219.11.254>
Jan 12 14:45:41 geonode sshd[52394]: Failed password for backup from 139.219.11.254 port 43450 ssh2
Jan 12 14:45:41 geonode sshd[52394]: Received disconnect from 139.219.11.254 port 43450:11: Bye Bye [preauth]
Jan 12 14:45:41 geonode sshd[52394]: Disconnected from authenticating user backup 139.219.11.254 port 43450 [preauth]
Jan 12 14:45:42 geonode sudo[52403]:  geonode : TTY=pts/1 ; PWD=/opt/geonode ; USER=root ; COMMAND=/usr/bin/journalctl -xe
Jan 12 14:45:42 geonode sudo[52403]: pam_unix(sudo:session): session opened for user root by geonode(uid=0)

[t-book]

Hi,

unsure if this can help somehow. I'm able to get the service running when disabling the pid file.

#pidfile = /tmp/geonode.pid

Afterward, I've restarted the service with

sudo rm /tmp/geonode.pid # just to be sure
sudo killall /usr/bin/uwsgi  # just to be sure
sudo service uwsgi restart

Unfortunately, I cannot say (yet) why this works. My first guess is it has to do with permissions. I've recognized that the pidfile in tmp is owned by root but the process runs with www-data, so maybe uwsgi is not able to recreate it. But this has to be checked (-> worth a try https://stackoverflow.com/a/25259447/1474777). Would be interesting if this solves it for others as well …

[gannebamm]

For me it does not work with a commented out pidfile. Here is my config:

https://gist.github.com/gannebamm/02289155c79f935b01efd355ee19d630

• commented pid file
• user tisdex / group www-data

But the service will not properly start and fail and restart in a loop.

If I use the shell script instead:

tisdex@zi-sws-004v-dmz:/opt/geonode$ ./geonode-uwsgi.sh
[uWSGI] getting INI configuration from /etc/uwsgi/apps-enabled/geonode.ini

It will start properly and the workers will get cheaped, like described above

[afabiani]

@gannebamm if the command works that means that the uwsgi is correct. Try maybe to play a bit with those parameters

SuccessExitStatus=143
TimeoutStopSec=10
Restart=on-success #on-failure
RestartSec=5

Remember that whenever you change that configuration you will need to run sudo systemctl daemon-reload

@jlsevillano as the error states you are missing geonode.local_settings. You can fix this by either:

a) Changing your uwsgi.ini by using geonode.settings instead of geonode.local_settings b) Creating a geonode.local_settings file accordingly

[gannebamm]

I switched to supervisord and it still fails. But I have some more info! It seems like the process is spawned multiple times, even though the first one is not ready yet. See the log:

https://gist.github.com/gannebamm/e5faadac13018bbdeb1568f7df5047a8 (updated the notes to be english instead of german)

The first spawned process will remain open and actually start to get cheaped, like described above. Nonetheless, since more recent processes do fail supervisor will report it as failed:

sudo supervisorctl start geonode_uwsgi
> geonode_uwsgi: ERROR (spawn error)

sudo supervisorctl status
> geonode_uwsgi                    FATAL     Exited too quickly (process log may have details)

[iamtekson]

I also faced the same problem when I tried to install geonode. After reading all the conservation here, I feel that the documentation is missing the geonode.local_settings.py file setup step. I think the following line needed to be mentioned before creating the app-available/geonode.ini file,

sudo cp /opt/geonode/geonode/local_settings.py.geoserver.sample /opt/geonode/geonode/local_settings.py

[iamtekson]

Hi @gannebamm, @afabiani I am still getting the error while restarting the uwsgi server. I checked the geonode log file, it shows the following error,

Fatal Python error: Py_Initialize: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

In my geonode.ini setting file, I have the following configuration, ( My ubuntu user is: administrator and venv name is: geonode)

gid = www-data
plugins = python3
virtualenv = /home/administrator/.virtualenvs/geonode

I think my virtualenv is not connected to geonode.ini setting or maybe the permission error. Any help, please

[allilou]

I used this service configuration file and it works

[Unit]
Description=GeoNode UWSGI Service

[Service]

ExecStartPre=/bin/bash -c 'mkdir -p /run/uwsgi; chown geonode:www-data /run/uwsgi'
ExecStart=uwsgi --emperor /etc/uwsgi/apps-enabled/geonode.ini

Restart=always
KillSignal=SIGQUIT
Type=notify
NotifyAccess=all

[Install]
WantedBy=multi-user.target

[villa101101]

I am also having similar issues

[iamtekson]

Hi @villa101101, For me, it was the conflict between python 3.6 and python 3.8. When I checked the python in the terminal, I got two versions and the 3.6 version was used by UWSGI. First of all, I uninstalled both version of python and installed python 3.6 instead of 3.8 and it worked. But according to the geonode documentation, we should install 3.8.

[thevinhhuynh]

my work is done, comment in "/etc/uwsgi/apps-available/geonode.ini"

pidfile = /tmp/geonode.pid

rm -rf /tmp/geonode.pid

sudo pkill -9 -f uwsgi

sudo service uwsgi restart

[iamtekson]

Thanks, @thevinhhuynh, It is the perfect solution. I think we need to update the geonode documentation as well.

[afabiani]

https://github.com/GeoNode/documentation/commit/f3ef8daa6a7c43550c262275bd704bbd74e8f394