Closed kougo-minami closed 4 months ago
Hi @kougo-minami,
According to the logs you shared, it looks like the services are not properly stopped.
bitnami.service: Unit process 766 (httpd) remains running after unit stopped.
bitnami.service: Unit process 772 (httpd) remains running after unit stopped.
bitnami.service: Unit process 1293 (httpd) remains running after unit stopped.
bitnami.service: Unit process 1427 (httpd) remains running after unit stopped.
bitnami.service: Unit process 1428 (httpd) remains running after unit stopped.
bitnami.service: Unit process 1753 (svnserve) remains running after unit stopped.
bitnami.service: Unit process 1765 (mysqld) remains running after unit stopped.
bitnami.service: Unit process 1907 (ruby) remains running after unit stopped.
Failed to start LSB: bitnami init script.
Let's try to manually stop them running this command
sudo /opt/bitnami/ctlscript.sh stop
Next check no Bitnami services are running using ps
, and kill any process running. After that, check the systemd service is stopped, and restart it.
Thanks for your confirmation.
sudo /opt/bitnami/ctlscript.sh stop
I executed the above command, but the status of bitnami.service remained failed.
It is said that all running processes will be forcibly terminated and then restarted, but is it correct to execute the following command? ・kill -9 -- -1 ・sudo /opt/bitnami/ctlscript.sh restart
Hi @kougo-minami,
Identify dangling processes with the command below and manually kill them. After that, try to restart the services. If it fails, please share both the command below and systemclt status
outputs
COLUMNS=9999 ps aux | grep -i bitnami
COLUMNS=9999 ps aux | grep -i bitnami
I ran the above command, but the dangling processes didn't seem to exist.
root 751 0.0 0.2 87776 11600 ? Ss Jan29 0:05 /opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf
daemon 775 0.0 0.2 2071708 10044 ? Sl Jan29 0:00 /opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf
daemon 776 0.0 0.3 2071708 12328 ? Sl Jan29 0:00 /opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf
daemon 777 0.0 0.2 2071708 11508 ? Sl Jan29 0:00 /opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf
daemon 782 0.0 0.3 2071708 13564 ? Sl Jan29 0:00 /opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf
daemon 1304 0.0 0.4 2071708 16236 ? Sl Jan29 0:00 /opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf
daemon 1438 0.0 0.3 2071708 13128 ? Sl Jan29 0:00 /opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf
daemon 1439 0.0 0.3 2071860 14924 ? Sl Jan29 0:00 /opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf
subvers+ 1764 0.0 0.0 17000 2724 ? Ss Jan29 0:00 svnserve -d --listen-host=0.0.0.0 --listen-port=3690 --pid-file=/opt/bitnami/subversion/tmp/svnserve.pid --log-file=/opt/bitnami/subversion/logs/svnserve.log
mysql 1776 0.0 10.9 1672044 441228 ? Sl Jan29 0:43 /opt/bitnami/mariadb/sbin/mysqld --defaults-file=/opt/bitnami/mariadb/conf/my.cnf --basedir=/opt/bitnami/mariadb --datadir=/bitnami/mariadb/data --socket=/opt/bitnami/mariadb/tmp/mysql.sock --pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid
daemon 1914 0.0 4.0 418484 163728 ? Sl Jan29 0:01 Passenger RubyApp: /opt/bitnami/redmine (production)
bitnami 34290 0.1 0.1 15148 7876 ? Ss 01:37 0:00 /lib/systemd/systemd --user
bitnami 34291 0.0 0.0 167612 2736 ? S 01:37 0:00 (sd-pam)
bitnami 34310 0.0 0.1 14276 4692 ? S 01:37 0:00 sshd: xxxxxxx@pts/0
bitnami 34311 0.0 0.1 8164 4956 pts/0 Ss 01:37 0:00 -bash
bitnami 34328 0.0 0.0 9756 3368 pts/0 R+ 01:38 0:00 ps aux
bitnami 34329 0.0 0.0 6372 708 pts/0 S+ 01:38 0:00 grep -i bitnami
However, compared to the normal Redmine server, the abnormal Redmine server seemed to lack the following processes.
Is this relevant?
root 1830 0.0 0.2 1678148 11340 ? Ssl 01:56 0:00 /opt/bitnami/gonit/bin/gonit
Hi @kougo-minami,
Please stop all services running the command below, and then check if there is any dangling service running the ps
command again. If you find any, please kill them, and then restart all services again.
sudo /opt/bitnami/ctlscript.sh stop
Regarding gonit, it is weird it is not running on your server. Did you modify anything on its configuration? Can you try to manually start it running the next command?
sudo gonit start
After that, verify it is running properly by running this other command
sudo gonit status
Hope it helps!
We have confirmed the following, so please check it.
【gonit】 ①Check Status(sudo gonit status) →The following error is output: "Connnot find any running daemon to contact. If it is running, make sure you are pointing to the right pid file(/var/run/gonit.pid)" ②Start ginit(sudo gonit start) →No Errors ③Check Status(sudo gonit status) →The following error is output: "Connnot find any running daemon to contact. If it is running, make sure you are pointing to the right pid file(/var/run/gonit.pid)" ④Check /var/run/gonit.pid →File does not exist cat /var/run/gonit.pid →cat: /var/run/gonit.pid: No such file or directory ls -la /var/run/gonit.pid →ls: cannot access '/var/run/gonit.pid': No such file or directory
【Process Restart】 ①Stop the process with the following command. sudo /opt/bitnami/ctlscript.sh stop ②The command in ① does not cause an error, but the process is not stopped. ③Manually stop the process(SSH etc. are not eligible.) ④The execution result of the following command remains as failed. systemctl status bitnami.service ⑤Confirm that browser access is no longer available. ⑥Start the process with the following command. sudo /opt/bitnami/ctlscript.sh start →I get the following error: Job for bitnami.service failed because the control process exited with error code. See "systemctl status bitnami.service" and "journalctl -xe" for details. ⑦Verify that the process is launched ⑧Verify that browser access is now available ⑨The execution result of the following command remains as failed systemctl status bitnami.service
Best regards.
Hi @kougo-minami,
It is weird that Gonit is not working on your server. It is also weird that start command returns errors but the app works on browser (so it is running on your server). Did you modify anything related to gonit? Please take a look at our docs
https://docs.bitnami.com/aws/apps/wordpress-multisite/administration/configure-use-gonit/
Please check the following.
【gonit】 1.When I executed the following command, gonit was able to start. "sudo gonit" 2.The status didn't seem to be a problem. "sudo gonit status"
Uptime 6m13s
Last Check 2024-02-06 06:04:37.677493744 +0000 UTC m=+360.084858590
Next Check 2024-02-06 06:06:37.677493744 +0000 UTC m=+480.084858590
Pid 226346
Pid File /var/run/gonit.pid
Control File /etc/gonit/gonitrc
Socket File /var/run/gonit.sock
Log File /var/log/gonit.log
Process 'apache'
status Running
pid 75741
uptime 6m13s
monitoring status monitored
Process 'mariadb'
status Running
pid 76795
uptime 6m13s
monitoring status monitored
Process 'subversion'
status Running
pid 76754
uptime 6m13s
monitoring status monitored
3."COLUMNS=9999 ps aux | grep -i bitnami" does not include gonit in the execution result, but we were able to confirm that gonit is started with the following command.
COLIMNS=9999 ps aux | grep 226346
root 226346 0.0 0.3 1678404 14832 ? Ssl 05:58 0:00 gonit
bitnami 229504 0.0 0.0 6240 708 pts/0 S+ 07:58 0:00 grep 226346
【bitnami.service】 1.After starting gonit, I checked the status of bitnami.sevice again, but it remained failed.
$systemctl status bitnami.service
● bitnami.service - LSB: bitnami init script
Loaded: loaded (/etc/init.d/bitnami; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2024-02-02 04:06:42 UTC; 4 days ago
Process: 75626 ExecStart=/etc/init.d/bitnami start (code=exited, status=1/FAILURE)
Tasks: 942 (limit: 4696)
Memory: 561.5M
CPU: 19min 11.998s
CGroup: /system.slice/bitnami.service
tq75741 /opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf
tq75745 Passenger watchdog
tq75748 Passenger core
tq75759 /opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf
tq75763 /opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf
tq75764 /opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf
tq75765 /opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf
tq76293 /opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf
tq76425 /opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf
tq76426 /opt/bitnami/apache/bin/httpd -f /opt/bitnami/apache/conf/httpd.conf
tq76754 svnserve -d --listen-host=0.0.0.0 --listen-port=3690 --pid-file=/opt/bitnami/subversion/tmp/svnserve.pid --log-file=/opt/bitnami/subver>
tq76762 Passenger RubyApp: /opt/bitnami/redmine (production)
mq76795 /opt/bitnami/mariadb/sbin/mysqld --defaults-file=/opt/bitnami/mariadb/conf/my.cnf --basedir=/opt/bitnami/mariadb --datadir=/bitnami/mar>
bitnami.service: Unit process 75765 (httpd) remains running after unit stopped.
bitnami.service: Unit process 76293 (httpd) remains running after unit stopped.
bitnami.service: Unit process 76425 (httpd) remains running after unit stopped.
bitnami.service: Unit process 76426 (httpd) remains running after unit stopped.
bitnami.service: Unit process 76691 (ruby) remains running after unit stopped.
bitnami.service: Unit process 76754 (svnserve) remains running after unit stopped.
bitnami.service: Unit process 76762 (ruby) remains running after unit stopped.
bitnami.service: Unit process 76795 (mysqld) remains running after unit stopped.
Failed to start LSB: bitnami init script.
bitnami.service: Consumed 5.815s CPU time.
2.I also ran the service startup command, but the result was the same.
$sudo /opt/bitnami/ctlscript.sh start
Starting services..
Job for bitnami.service failed because the control process exited with error code.
See "systemctl status bitnami.service" and "journalctl -xe" for details.
Thank you for your support.
Hi @kougo-minami,
Thanks for sharing the info. It is great gonit is working now. Regarding the systemctl service error messages, I think although the systemctl stop command stops the services, there are processes still running (although you mentioned that COLUMNS=9999 ps aux | grep -i bitnami
doesn't print any process), or the PID caching file for those services are not properly removed. Please stop all services, including gonit to ensure it doesn't start them in the background, then check if any process from Bitnami folder is running with ps. After that, kill any process found, and also the processes from the PIDs detected by the systemctl status
command. Also look for the .pid
files under the /opt/bitnami
directory and remove them (if any). That should properly stop all running services. Finally, start Bitnami services again and check service status.
The following steps have been taken, but the situation remains the same. Please let me know if there is anything else I should check.
sudo /opt/bitnami/ctlscript.sh stop →Stop the process COLUMNS=9999 ps aux | grep -i bitnami COLIMNS=9999 ps aux | grep -i gonit →Confirm bitnami-related & gonit process ID sudo kill [PID] →Forcibly stop processes that could not be stopped find /opt/bitnami -iname "*.pid" →Search for pid file sudo rm [pid file] →Delete the remaining pid file systemctl status bitnami.service →Check the status of bitnami service (it was still "failed") sudo /opt/bitnami/ctlscript.sh start →Start the process (same error as before) COLUMNS=9999 ps aux | grep -i bitnami COLIMNS=9999 ps aux | grep -i gonit →Check the status of bitnami-related process startup (it has been started) systemctl status bitnami.service →Check the status of bitnami service (it was still "failed")
Hi @kougo-minami,
I'm afraid I'm running out of ideas here. Can you follow the same process you described but use systemctl
command to stop/start the Bitnami service instead of calling the ctlscript directly?
In case the above doesn't work, we recommend you to start a new instance and migrate your application data there. Sorry for the inconveniences.
I tried to start and stop the bitnami service using the systemctl command with the same result. I tried to rebuild the redmine server, but the bitnami service was still FAILED from the initial state. (I rebuilt the server once before, and the same situation happened then.) I can set up a redmine server in another Azure environment (different subscription) without any problem.
Currently, the status of the bitnami service is failed, but browser access is working fine. However, is it correct to say that it is not recommended to operate the bitnami service in production with the status of "failed"?
Hi @kougo-minami,
We recommend you to start a new server and migrate the Redmine data there. Although the web application in working on your current server, I think it is better to have a fully working instance, but it is a decision you have to make having your whole scenario into account. Hope it helps!
I have started a new server (bitnami package for redmine -x64 Gen1) twice and both times bitnami.service was initially failed just like the current server. Is it better to start a Windows server and migrate redmine to it instead of "bitnami package for redmine -x64 Gen1"?
Hi @kougo-minami,
Can you tell me the exact Redmine version and machine size you are using so I can try to reproduce your issue to take a look at it?
This will be the information on the machine where bitnami.service becomes FAILED. Version: 5.1.1 Size: Standard B2s
Thank you in advance.
Hi @kougo-minami,
I have launched an standard B2s instance with Redmine 5.1.2 in Azure, which is the current version available. I have not been able to launch any previous version. I didn't find any issue with the initialization process of this machine, and the Bitnami service is properly running right after the initialization process has finished according to the systemctl output
$ sudo systemctl status bitnami
● bitnami.service - LSB: bitnami init script
Loaded: loaded (/etc/init.d/bitnami; enabled; preset: enabled)
Active: active (running) since Thu 2024-03-14 09:56:40 UTC; 10s ago
Process: 7539 ExecStart=/etc/init.d/bitnami start (code=exited, status=0/SUCCESS)
Tasks: 958 (limit: 4691)
Memory: 704.6M
CPU: 7.548s
However, I have been able to reproduce your issue by force killing gonit PID. After that, the Bitnami service showed the same failed status you were getting. I have fixed it by starting gonit and telling it to monitor the Bitnami services again. Can you run the next commands on your server and share the output with us?
sudo systemctl status
sudo ps aux | grep -i gonit
sudo gonit status
sudo rm -vf /var/run/gonit.pid /var/run/gonit.sock
sudo gonit
sudo gonit status
sudo gonit monitor apache
sudo gonit monitor mariadb
sudo gonit monitor subversion
sudo gonit status
sudo systemctl restart bitnami
sudo systemctl status bitnami
journalctl -u bitnami
Hope it helps!
Thank you for your response. We will share the results of the execution of the command you linked to us in a text file.
Please check it out.
Hi @kougo-minami,
Thanks for your message. I'm glad you could start the Gonit monitor. However, it is weird that there are still processes running after the systemd service has been stopped. Can you please run the next commands to stop the services and manually kill the pending processes (notice I'm using the PIDs from the latest lines in the journalctl command
)
curl -Li "http://127.0.0.1"
sudo free -m
uptime
sudo netstat -tupln
sudo /opt/bitnami/ctlscript.sh stop
sudo systemctl stop bitnami
sudo kill -9 762 766 769 779 782 783 791 1314 1448 1449 1712 1774 1787 1908 1928
sudo systemctl status bitnami
sudo netstat -tupln
sudo systemctl start bitnami
sudo systemctl status bitnami
sudo journalctl -u bitnami | tail -n 25
curl -Li "http://127.0.0.1"
This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.
Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.
Platform
Azure
bndiagnostic ID know more about bndiagnostic ID
Automatic upload to Bitnami servers failed.
bndiagnostic output
===== Begin of bndiagnostic tool output =====
[Connectivity]
Server ports 22, 80 and/or 443 are not publicly accessible. Please check the following guide to open server ports for remote access:
https://docs.bitnami.com/general/faq/administration/use-firewall/
[Apache]
Found recent error or warning messages in the Apache error log.
Please check the following guide to troubleshoot server issues:
https://docs.bitnami.com/general/apps/wordpress/troubleshooting/debug-erro rs-apache/
===== End of bndiagnostic tool output =====
bndiagnostic was not useful. Could you please tell us why?
Automatic upload to Bitnami servers failed.
Describe your issue as much as you can
We built a bitnami server with "bitnami package for redmine -x64 Gen1" on Azure. When accessed with a browser in the initial construction state, the login screen is displayed without any problem, but when "systemctl status bitnami.service" is executed, the status of bitnami.service is failed. Is it okay for bitnami.service to be in the failed state? If there is a problem, please let me know how to solve it.
Here are some of the things I tried to activate the status.
・Run "sudo /opt/bitnami/ctlscript.sh restart" →The following error is returned: Job for bitnami.service failed because the control process exited with error code. See "systemctl status bitnami.service" and "journalctl -xe" for details.
・The result of "systemctl status bitnami.service" is as follows.
・The result of "journalctl -xe" is as follows.
・/opt/bitnami/redmine/log/production.log
・/opt/bitnami/apache2/logs/error_log