bitnami / vms

Bitnami VMs
https://bitnami.com
Other
203 stars 42 forks source link

[<aws lightsail bitnami wordpress>] <bn diagnostic found some issue> #1382

Closed livewithsjsj closed 5 months ago

livewithsjsj commented 6 months ago

Platform

AWS

bndiagnostic ID know more about bndiagnostic ID

ff0bb617-6fbb-ad0e-fc38-7bcb7ce53677

bndiagnostic output

? Resources: Found possible issues
? Connectivity: Found possible issues
✓ Mariadb: No issues found
✓ Processes: No issues found
✓ Wordpress: No issues found
? Apache: Found possible issues
✓ Php: No issues found

[Resources] Your instance has little available RAM memory.

 total used free shared buff/cache available Mem: 15825 11528 1881 157 2414 3813 
Swap: 0 0 0

You could try to increase your instance's memory. Please check your cloud provider's documentation for more information. Press [Enter] to continue:

You can also enable swap memory to improve performance.

https://docs.bitnami.com/installer/faq/linux-faq/administration/increase-m emory-linux/

[Connectivity]

Server ports 22, 80 and/or 443 are not publicly accessible. Please check the following guide to open server ports for remote access:

https://docs.bitnami.com/general/faq/administration/use-firewall/

[Apache]

Found recent error or warning messages in the Apache error log.

[Tue Jan 23 12:05:17.943805 2024] [access_compat:error] [pid 16078:tid 
140634123917056] [client **ip_address**:64493] AH01797: client denied by server 
configuration: /opt/bitnami/apache/htdocs/
 [Tue Jan 23 12:29:23.313920 2024] [access_compat:error] [pid 16078:tid 
140634761434880] [client **ip_address**:51518] AH01797: client denied by server 
configuration: /opt/bitnami/apache/htdocs/
Press [Enter] to continue:
 [Tue Jan 23 12:34:40.251449 2024] [proxy_fcgi:error] [pid 16062:tid 
140636254549760] [client **ip_address**:37666] AH01071: Got error 'PHP message: 
woocommerce_add_order_item_meta is deprecated since version 3.0.0! Use 
woocommerce_new_order_item instead.; PHP message: 
woocommerce_add_order_item_meta is deprecated since version 3.0.0! Use 
woocommerce_new_order_item instead.; PHP message: 
woocommerce_add_order_item_meta is deprecated since version 3.0.0! Use 
woocommerce_new_order_item instead.', 

Please check the following guide to troubleshoot server issues:

https://docs.bitnami.com/general/apps/wordpress/troubleshooting/debug-erro rs-apache/

bndiagnostic was not useful. Could you please tell us why?

useful

Describe your issue as much as you can

hello. My homepage is running using bitnami wordpress on aws lightsail. Thank you Bitnami!

These days, when the CPU overview goes up to about 20%, the homepage suddenly crashes. So I tried running bndiagnotic-tool.

  1. little available Ram memory Bndiagnotic first says there is not enough RAM, but I am using 16GB RAM, 4v CPUs, and 320GB SSD. Normally, CPU overview is less than 10%. But I don't know why it is diagnosed as having insufficient RAM. In wp-config, the memory is set to use up to 1024MB. In php.ini, it is set to use up to 512MB.

2.connectivity diagnosis https://docs.bitnami.com/general/faq/administration/use-firewall/ I read the above manual, and it appears to open a port for remote access. Currently, my server's IPv4 Firewell settings are SSH-port22, HTTP-port80, and HTTPS-port443. Is there anything I should add?

3.Apache diagnosis There were three warning messages. There were two client denied by server and got error 'php message'. The client IP addresses in the three messages are all different. Can this be considered an attack?

In conclusion, I don't know why aws lightsail bitnami wordpress keeps turning off when CPU usage exceeds 20%-25%. If I reboot the server, it will operate normally again immediately.

I would really appreciate your help.

livewithsjsj commented 6 months ago

For reference, the amount of memory checked by WordPress's query monitor plugin and the server (free -m) is as follows.

-query monitor - memory check- Peak Memory Usage 80,121,912 bytes (76.4 MB) 7.5% of 1,024 MB server limit

-aws server - memory check- total used free shared buff/cache available 15825 13215 285 158 2324 2127 Swap: 0(total) 0(used) 0(free)

gongomgra commented 6 months ago

Hi @livewithsjsj,

Thanks for using Bitnami. The bndiagnostic tool internally considers that if your server has less than 20% of free memory available, you may run into issues starting new processes. However, it is not your case, as your server has enough memory available

-----------------------------------
Display amount of free and used memory in the system
-----------------------------------
Running: free -m
In: /opt/bitnami

Output:

               total        used        free      shared  buff/cache   available
Mem:           15825       11296        1980         157        2548        4046
Swap:              0           0           0

About the CPU consumption, I see you have lots of requests from different IP addresses. Unfortunately, I don't know if those are legit IP addresses or not, but you can use our guide on how to deny connections from bots/attackers using Apache to detect and block them

-----------------------------------
Check performance issues: Count number of requests for the 10 most active IP addresses in the last 100.000 requests
-----------------------------------
Running: tail -n 100000 access_log | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 10 | awk '{print $1}'
In: /opt/bitnami/apache2/logs/

Output:

2553
1324
1086
1058
985
938
876
831
827
817

Additionally, Lightsail instances are of type burstable, which consume CPU credits over time and reduces performance if you run out of them. Credits can be rapidly consumed in high traffic websites, or if the website has a lot of interaction with its database (like plugins saving/extracting internal data). Blocking bots (if any) will reduce the CPU usage, but it may not be enough depending on your scenario.

Hope it helps!

livewithsjsj commented 6 months ago

Hello @gongomgra , thank you for the quick reply.

  1. I understand about memory usage. I don’t think there will be any major issues if I keep it as is!
  2. All IP addresses have been confirmed to be customer IP addresses. We are also working on blocks through jetpack and wordfence.
  3. There is plenty of CPU left. CPU credit is also maintained at almost 100%.

Nonetheless, sometimes the homepage crashes. If I reboot the server, it will be able to connect again. In general, this phenomenon seems to occur when CPU usage is over 20% (CPU credits remain over 80%).

So, I tried diagnosing it using bndiagnostic. I wonder what the reason is. At first I thought it was a memory issue, but if not a memory issue, what could be the problem?

iba-1 commented 6 months ago

@gongomgra a really long series of issues opened in this repo seems to point at some kind of memory leak in this type of image, maybe we should look into it, can you point us somewhere? Have you had any idea?

gongomgra commented 6 months ago

Hi @iba-1, @livewithsjsj,

We have found a lot of performance issues in the past due to burstable instances. We also updated PHP memory settings recently (see issues/927). Unfortunately, Lightsail team usually takes some extra time to publish new images. You may want to manually update memory settings as decribed in that thread and restart services for changes to take effect.

Hope it helps!

livewithsjsj commented 6 months ago

Hi @gongomgra

Then... I have to update it manually. So how do I update it? Is there a manual in issue/927? Please note that my homepage is not a WordPress multisite. This is a single site. It would be even better if there was an organized manual.

livewithsjsj commented 6 months ago

@gongomgra

I read issue/927. Are the things below that I need to change manually? Are you saying that I have to change everything to fit the details below? After checking, please give me some advice. I am also curious about where to find the values below and how to change them. thank you

# ./memory-micro.conf
; Bitnami memory configuration for PHP-FPM
;
; Note: This will be modified on server size changes

pm.max_children=5
pm.start_servers=1
pm.min_spare_servers=1
pm.max_spare_servers=3
pm.max_requests=5000

# ./memory-small.conf
; Bitnami memory configuration for PHP-FPM
;
; Note: This will be modified on server size changes

pm.max_children=10
pm.start_servers=2
pm.min_spare_servers=2
pm.max_spare_servers=5
pm.max_requests=5000

# ./memory-medium.conf
; Bitnami memory configuration for PHP-FPM
;
; Note: This will be modified on server size changes

pm.max_children=25
pm.start_servers=4
pm.min_spare_servers=4
pm.max_spare_servers=10
pm.max_requests=5000

# ./memory-large.conf
; Bitnami memory configuration for PHP-FPM
;
; Note: This will be modified on server size changes

pm.max_children=50
pm.start_servers=5
pm.min_spare_servers=5
pm.max_spare_servers=30
pm.max_requests=5000

# ./memory-xlarge.conf
; Bitnami memory configuration for PHP-FPM
;
; Note: This will be modified on server size changes

pm.max_children=125
pm.start_servers=6
pm.min_spare_servers=6
pm.max_spare_servers=50
pm.max_requests=5000

# ./memory-2xlarge.conf
; Bitnami memory configuration for PHP-FPM
;
; Note: This will be modified on server size changes

pm.max_children=250
pm.start_servers=7
pm.min_spare_servers=7
pm.max_spare_servers=100
pm.max_requests=5000
gongomgra commented 6 months ago

Hi @livewithsjsj,

You only need to update the PHP-FPM settings (located at /opt/bitnami/php/etc/memory.conf) with the values corresponding to the size of your server and then restart services for changes to take effect. You can disregard the rest of the changes, unless you are planing to use another instance size.

livewithsjsj commented 6 months ago

@gongomgra I am using Lightsail 16GB RAM, 4v CPU, 320GB SSD. Which size is right for me? The current PHP-FPM related values in memory.conf(located at /opt/bitnami/php/etc/memory.conf) are as follows.

; Bitnami memory configuration for PHP-FPM
;
; Note: This will be modified on server size changes

pm.max_children=200
pm.start_servers=130
pm.min_spare_servers=130
pm.max_spare_servers=150
pm.max_requests=5000

and

/opt/bitnami/mariadb/conf/bitnami/memory.conf

Is there any need to change the above file? The current MariaDB memory.conf(located at /opt/bitnami/mariadb/conf/bitnami/memory.conf) values are as follows.

[mysqld]
#wait_timeout = 120
long_query_time = 1
query_cache_size=0
innodb_buffer_pool_size=2048M
#innodb_log_file_size=128M
#innodb_flush_method=O_DIRECT
#tmp_table_size=64M
#max_connections = 2500
#max_user_connections = 2500
#key_buffer_size=64M

For reference, I have only opened one Lightsail instance. A separate database was not opened by Amazon.(In other words, PHP and MariaDB server software run on one machine.)

gongomgra commented 6 months ago

Hi @livewithsjsj,

Thanks for your message. Your current values correspond to the old values for the xlarge instance, so you can update the values to the ones below

# ./memory-xlarge.conf
; Bitnami memory configuration for PHP-FPM
;
; Note: This will be modified on server size changes

pm.max_children=125
pm.start_servers=6
pm.min_spare_servers=6
pm.max_spare_servers=50
pm.max_requests=5000

After that, restart all services for changes to take effect. Hope it helps!

livewithsjsj commented 6 months ago

hi @gongomgra As you mentioned above, we have changed our settings.

prev.

-aws server - memory check-
total used free shared buff/cache available
15825 13215 285 158 2324 2127
Swap: 0(total) 0(used) 0(free)

after change memory.conf setting

  total        used        free      shared  buff/cache   available
Mem:           15825       11244        2389         156        2191        4101

Swap:              0(total)           0(used)           0(free)

It still uses 10000 units of memory. T.T is it okay?

gongomgra commented 5 months ago

Hi @livewithsjsj,

Thanks for your message. Notice the overall memory usage is lower after updating the PHP-FPM settings, but the running processes (some of them also attending requests for your website) still consume memory. According to the latest free command output there are around 2 GB of free memory on your server, so I don't think you need to worry about that.

You can continue checking the list of processes consuming CPU and memory and other performance aspects by following the guide below

https://docs.bitnami.com/aws/faq/troubleshooting/troubleshoot-server-performance/

github-actions[bot] commented 5 months ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] commented 5 months ago

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.