search

matomo-org / matomo

Empowering People Ethically with the leading open source alternative to Google Analytics that gives you full control over your data. Matomo lets you easily collect data from websites & apps and visualise this data and extract insights. Privacy is built-in. Liberating Web Analytics. Star us on Github? +1. And we love Pull Requests!
https://matomo.org/
GNU General Public License v3.0
19.7k stars 2.62k forks source link

php-fpm segfault #20290

Closed ghost closed 1 year ago

ghost commented 1 year ago

Expected Behavior

No segfault

Current Behavior

At random intervals, matomo stops working. I can login, but I can't see any results in the dashboard, all attempts to process requests via the matomo-php-tracker all result in php-fpm segfaults, as seen below:

php-fpm[2355]: segfault at 7f10c583abd0 ip 0000557eeb067858 sp 00007ffeef69ec40 error 4 in php-fpm[557eead8f000+3f3000]
Code: f8 10 0f 84 aa 01 00 00 49 8b 46 18 f6 40 50 01 0f 85 94 00 00 00 49 8b 46 18 48 8b 40 50 48 8b 00 48 85 c0 0f 84 a0 00 00 00 <8b> 10 83 fa 01 76 2c f6 40 04 40 75 05 83 ea 01 89 10 48 89 c7 e8
php-fpm[2502]: segfault at 7f10c6145150 ip 0000557eeb067858 sp 00007ffeef69ec40 error 4 in php-fpm[557eead8f000+3f3000]
Code: f8 10 0f 84 aa 01 00 00 49 8b 46 18 f6 40 50 01 0f 85 94 00 00 00 49 8b 46 18 48 8b 40 50 48 8b 00 48 85 c0 0f 84 a0 00 00 00 <8b> 10 83 fa 01 76 2c f6 40 04 40 75 05 83 ea 01 89 10 48 89 c7 e8
php-fpm[2357]: segfault at 8 ip 0000557eeb0678a1 sp 00007ffeef69ec40 error 4 in php-fpm[557eead8f000+3f3000]
Code: 42 50 01 0f 85 93 00 00 00 49 8b 56 18 48 8b 52 50 48 89 02 41 8b 57 14 89 d5 83 e5 fc 48 03 68 10 83 e2 01 0f 84 8f 00 00 00 <0f> b6 45 08 3c 0b 0f 84 c3 00 00 00 3c 0a 0f 85 e3 00 00 00 48 8b
php-fpm[2643]: segfault at 8 ip 0000557eeb0678a1 sp 00007ffeef69ec40 error 4 in php-fpm[557eead8f000+3f3000]
Code: 42 50 01 0f 85 93 00 00 00 49 8b 56 18 48 8b 52 50 48 89 02 41 8b 57 14 89 d5 83 e5 fc 48 03 68 10 83 e2 01 0f 84 8f 00 00 00 <0f> b6 45 08 3c 0b 0f 84 c3 00 00 00 3c 0a 0f 85 e3 00 00 00 48 8b
php-fpm[2358]: segfault at 7f10c6145150 ip 0000557eeb067868 sp 00007ffeef69ec40 error 7 in php-fpm[557eead8f000+3f3000]
Code: 0f 85 94 00 00 00 49 8b 46 18 48 8b 40 50 48 8b 00 48 85 c0 0f 84 a0 00 00 00 8b 10 83 fa 01 76 2c f6 40 04 40 75 05 83 ea 01 <89> 10 48 89 c7 e8 4e 9e fc ff 49 8b 56 18 f6 42 50 01 0f 85 93 00
php-fpm[2503]: segfault at 8 ip 0000557eeb0678a1 sp 00007ffeef69ec40 error 4 in php-fpm[557eead8f000+3f3000]
Code: 42 50 01 0f 85 93 00 00 00 49 8b 56 18 48 8b 52 50 48 89 02 41 8b 57 14 89 d5 83 e5 fc 48 03 68 10 83 e2 01 0f 84 8f 00 00 00 <0f> b6 45 08 3c 0b 0f 84 c3 00 00 00 3c 0a 0f 85 e3 00 00 00 48 8b
php-fpm[2367]: segfault at 8 ip 0000557eeb0678a1 sp 00007ffeef69ec40 error 4 in php-fpm[557eead8f000+3f3000]
Code: 42 50 01 0f 85 93 00 00 00 49 8b 56 18 48 8b 52 50 48 89 02 41 8b 57 14 89 d5 83 e5 fc 48 03 68 10 83 e2 01 0f 84 8f 00 00 00 <0f> b6 45 08 3c 0b 0f 84 c3 00 00 00 3c 0a 0f 85 e3 00 00 00 48 8b
php-fpm[167398]: segfault at 8 ip 0000557eeb0678a1 sp 00007ffeef69ec40 error 4 in php-fpm[557eead8f000+3f3000]
Code: 42 50 01 0f 85 93 00 00 00 49 8b 56 18 48 8b 52 50 48 89 02 41 8b 57 14 89 d5 83 e5 fc 48 03 68 10 83 e2 01 0f 84 8f 00 00 00 <0f> b6 45 08 3c 0b 0f 84 c3 00 00 00 3c 0a 0f 85 e3 00 00 00 48 8b
php-fpm[2351]: segfault at 8 ip 0000557eeb0678a1 sp 00007ffeef69ec40 error 4 in php-fpm[557eead8f000+3f3000]
Code: 42 50 01 0f 85 93 00 00 00 49 8b 56 18 48 8b 52 50 48 89 02 41 8b 57 14 89 d5 83 e5 fc 48 03 68 10 83 e2 01 0f 84 8f 00 00 00 <0f> b6 45 08 3c 0b 0f 84 c3 00 00 00 3c 0a 0f 85 e3 00 00 00 48 8b
php-fpm[167400]: segfault at 8 ip 0000557eeb0678a1 sp 00007ffeef69ec40 error 4 in php-fpm[557eead8f000+3f3000]
Code: 42 50 01 0f 85 93 00 00 00 49 8b 56 18 48 8b 52 50 48 89 02 41 8b 57 14 89 d5 83 e5 fc 48 03 68 10 83 e2 01 0f 84 8f 00 00 00 <0f> b6 45 08 3c 0b 0f 84 c3 00 00 00 3c 0a 0f 85 e3 00 00 00 48 8b
php-fpm[167045]: segfault at 7f10c6145150 ip 0000557eeb067858 sp 00007ffeef69ec40 error 4 in php-fpm[557eead8f000+3f3000]
Code: f8 10 0f 84 aa 01 00 00 49 8b 46 18 f6 40 50 01 0f 85 94 00 00 00 49 8b 46 18 48 8b 40 50 48 8b 00 48 85 c0 0f 84 a0 00 00 00 <8b> 10 83 fa 01 76 2c f6 40 04 40 75 05 83 ea 01 89 10 48 89 c7 e8
php-fpm[2501]: segfault at 8 ip 0000557eeb0678a1 sp 00007ffeef69ec40 error 4 in php-fpm[557eead8f000+3f3000]
Code: 42 50 01 0f 85 93 00 00 00 49 8b 56 18 48 8b 52 50 48 89 02 41 8b 57 14 89 d5 83 e5 fc 48 03 68 10 83 e2 01 0f 84 8f 00 00 00 <0f> b6 45 08 3c 0b 0f 84 c3 00 00 00 3c 0a 0f 85 e3 00 00 00 48 8b
php-fpm[2356]: segfault at 8 ip 0000557eeb0678a1 sp 00007ffeef69ec40 error 4 in php-fpm[557eead8f000+3f3000]
Code: 42 50 01 0f 85 93 00 00 00 49 8b 56 18 48 8b 52 50 48 89 02 41 8b 57 14 89 d5 83 e5 fc 48 03 68 10 83 e2 01 0f 84 8f 00 00 00 <0f> b6 45 08 3c 0b 0f 84 c3 00 00 00 3c 0a 0f 85 e3 00 00 00 48 8b
php-fpm[167405]: segfault at 8 ip 0000557eeb0678a1 sp 00007ffeef69ec40 error 4 in php-fpm[557eead8f000+3f3000]
Code: 42 50 01 0f 85 93 00 00 00 49 8b 56 18 48 8b 52 50 48 89 02 41 8b 57 14 89 d5 83 e5 fc 48 03 68 10 83 e2 01 0f 84 8f 00 00 00 <0f> b6 45 08 3c 0b 0f 84 c3 00 00 00 3c 0a 0f 85 e3 00 00 00 48 8b
php-fpm[2365]: segfault at 8 ip 0000557eeb0678a1 sp 00007ffeef69ec40 error 4 in php-fpm[557eead8f000+3f3000]
Code: 42 50 01 0f 85 93 00 00 00 49 8b 56 18 48 8b 52 50 48 89 02 41 8b 57 14 89 d5 83 e5 fc 48 03 68 10 83 e2 01 0f 84 8f 00 00 00 <0f> b6 45 08 3c 0b 0f 84 c3 00 00 00 3c 0a 0f 85 e3 00 00 00 48 8b
 php-fpm[169450]: segfault at 7fe80d8f50e0 ip 0000559fa9214858 sp 00007ffdf392db20 error 4 in php-fpm[559fa8f3c000+3f3000]
 Code: f8 10 0f 84 aa 01 00 00 49 8b 46 18 f6 40 50 01 0f 85 94 00 00 00 49 8b 46 18 48 8b 40 50 48 8b 00 48 85 c0 0f 84 a0 00 00 00 <8b> 10 83 fa 01 76 2c f6 40 04 40 75 05 83 ea 01 89 10 48 89 c7 e8
 php-fpm[169406]: segfault at 7fe80d8f50e0 ip 0000559fa9214858 sp 00007ffdf392db20 error 4 in php-fpm[559fa8f3c000+3f3000]
 Code: f8 10 0f 84 aa 01 00 00 49 8b 46 18 f6 40 50 01 0f 85 94 00 00 00 49 8b 46 18 48 8b 40 50 48 8b 00 48 85 c0 0f 84 a0 00 00 00 <8b> 10 83 fa 01 76 2c f6 40 04 40 75 05 83 ea 01 89 10 48 89 c7 e8
 php-fpm[169448]: segfault at 7fe80e331620 ip 0000559fa9214858 sp 00007ffdf392db20 error 4 in php-fpm[559fa8f3c000+3f3000]
 Code: f8 10 0f 84 aa 01 00 00 49 8b 46 18 f6 40 50 01 0f 85 94 00 00 00 49 8b 46 18 48 8b 40 50 48 8b 00 48 85 c0 0f 84 a0 00 00 00 <8b> 10 83 fa 01 76 2c f6 40 04 40 75 05 83 ea 01 89 10 48 89 c7 e8
 php-fpm[169452]: segfault at 7fe80e331620 ip 0000559fa9214858 sp 00007ffdf392db20 error 4 in php-fpm[559fa8f3c000+3f3000]
 Code: f8 10 0f 84 aa 01 00 00 49 8b 46 18 f6 40 50 01 0f 85 94 00 00 00 49 8b 46 18 48 8b 40 50 48 8b 00 48 85 c0 0f 84 a0 00 00 00 <8b> 10 83 fa 01 76 2c f6 40 04 40 75 05 83 ea 01 89 10 48 89 c7 e8
 php-fpm[169455]: segfault at 14600000001 ip 0000559fa9254c87 sp 00007ffdf392db40 error 6 in php-fpm[559fa8f3c000+3f3000]
 Code: 66 0f ef c0 f2 0f 10 0f f2 48 0f 2a 06 e9 a8 e7 ff ff e8 dc 7f fc ff 41 ff 27 8b 50 10 48 8b 48 08 f6 c6 ff 0f 84 45 02 00 00 <83> 01 01 8b 50 10 49 83 c7 20 48 8b 40 08 89 56 08 48 89 06 41 ff
 traps: php-fpm[169416] trap stack segment ip:559fa92148a1 sp:7ffdf392db20 error:0 in php-fpm[559fa8f3c000+3f3000]
 php-fpm[174544]: segfault at 7fe80e535508 ip 0000559fa9214858 sp 00007ffdf392db20 error 4 in php-fpm[559fa8f3c000+3f3000]
 Code: f8 10 0f 84 aa 01 00 00 49 8b 46 18 f6 40 50 01 0f 85 94 00 00 00 49 8b 46 18 48 8b 40 50 48 8b 00 48 85 c0 0f 84 a0 00 00 00 <8b> 10 83 fa 01 76 2c f6 40 04 40 75 05 83 ea 01 89 10 48 89 c7 e8
 php-fpm[169411]: segfault at 7fe80e535508 ip 0000559fa9214858 sp 00007ffdf392db20 error 4 in php-fpm[559fa8f3c000+3f3000]
 Code: f8 10 0f 84 aa 01 00 00 49 8b 46 18 f6 40 50 01 0f 85 94 00 00 00 49 8b 46 18 48 8b 40 50 48 8b 00 48 85 c0 0f 84 a0 00 00 00 <8b> 10 83 fa 01 76 2c f6 40 04 40 75 05 83 ea 01 89 10 48 89 c7 e8
 php-fpm[169422]: segfault at 7fe80e535508 ip 0000559fa9214858 sp 00007ffdf392db20 error 4 in php-fpm[559fa8f3c000+3f3000]
 Code: f8 10 0f 84 aa 01 00 00 49 8b 46 18 f6 40 50 01 0f 85 94 00 00 00 49 8b 46 18 48 8b 40 50 48 8b 00 48 85 c0 0f 84 a0 00 00 00 <8b> 10 83 fa 01 76 2c f6 40 04 40 75 05 83 ea 01 89 10 48 89 c7 e8
 php-fpm[169412]: segfault at 7fe80e535508 ip 0000559fa9214858 sp 00007ffdf392db20 error 4 in php-fpm[559fa8f3c000+3f3000]
 Code: f8 10 0f 84 aa 01 00 00 49 8b 46 18 f6 40 50 01 0f 85 94 00 00 00 49 8b 46 18 48 8b 40 50 48 8b 00 48 85 c0 0f 84 a0 00 00 00 <8b> 10 83 fa 01 76 2c f6 40 04 40 75 05 83 ea 01 89 10 48 89 c7 e8
 php-fpm[169439]: segfault at 7fe80e535508 ip 0000559fa9214858 sp 00007ffdf392db20 error 4 in php-fpm[559fa8f3c000+3f3000]
 Code: f8 10 0f 84 aa 01 00 00 49 8b 46 18 f6 40 50 01 0f 85 94 00 00 00 49 8b 46 18 48 8b 40 50 48 8b 00 48 85 c0 0f 84 a0 00 00 00 <8b> 10 83 fa 01 76 2c f6 40 04 40 75 05 83 ea 01 89 10 48 89 c7 e8
 php-fpm[169444]: segfault at 8 ip 0000559fa92148a1 sp 00007ffdf392db20 error 4 in php-fpm[559fa8f3c000+3f3000]
 Code: 42 50 01 0f 85 93 00 00 00 49 8b 56 18 48 8b 52 50 48 89 02 41 8b 57 14 89 d5 83 e5 fc 48 03 68 10 83 e2 01 0f 84 8f 00 00 00 <0f> b6 45 08 3c 0b 0f 84 c3 00 00 00 3c 0a 0f 85 e3 00 00 00 48 8b
 php-fpm[174771]: segfault at 7fe80e535508 ip 0000559fa9214858 sp 00007ffdf392db20 error 4 in php-fpm[559fa8f3c000+3f3000]
 Code: f8 10 0f 84 aa 01 00 00 49 8b 46 18 f6 40 50 01 0f 85 94 00 00 00 49 8b 46 18 48 8b 40 50 48 8b 00 48 85 c0 0f 84 a0 00 00 00 <8b> 10 83 fa 01 76 2c f6 40 04 40 75 05 83 ea 01 89 10 48 89 c7 e8
 php-fpm[169425]: segfault at 7fe80e535508 ip 0000559fa9214858 sp 00007ffdf392db20 error 4 in php-fpm[559fa8f3c000+3f3000]
 Code: f8 10 0f 84 aa 01 00 00 49 8b 46 18 f6 40 50 01 0f 85 94 00 00 00 49 8b 46 18 48 8b 40 50 48 8b 00 48 85 c0 0f 84 a0 00 00 00 <8b> 10 83 fa 01 76 2c f6 40 04 40 75 05 83 ea 01 89 10 48 89 c7 e8
 traps: php-fpm[169430] trap stack segment ip:559fa92148a1 sp:7ffdf392db20 error:0 in php-fpm[559fa8f3c000+3f3000]
 php-fpm[169433]: segfault at 8 ip 0000559fa92148a1 sp 00007ffdf392db20 error 4 in php-fpm[559fa8f3c000+3f3000]
 Code: 42 50 01 0f 85 93 00 00 00 49 8b 56 18 48 8b 52 50 48 89 02 41 8b 57 14 89 d5 83 e5 fc 48 03 68 10 83 e2 01 0f 84 8f 00 00 00 <0f> b6 45 08 3c 0b 0f 84 c3 00 00 00 3c 0a 0f 85 e3 00 00 00 48 8b
 php-fpm[169446]: segfault at 7fe80e535508 ip 0000559fa9214858 sp 00007ffdf392db20 error 4 in php-fpm[559fa8f3c000+3f3000]
 Code: f8 10 0f 84 aa 01 00 00 49 8b 46 18 f6 40 50 01 0f 85 94 00 00 00 49 8b 46 18 48 8b 40 50 48 8b 00 48 85 c0 0f 84 a0 00 00 00 <8b> 10 83 fa 01 76 2c f6 40 04 40 75 05 83 ea 01 89 10 48 89 c7 e8
 php-fpm[169414]: segfault at 559fa92bfb40 ip 0000559fa92149c0 sp 00007ffdf392db20 error 7 in php-fpm[559fa8f3c000+3f3000]
 Code: 00 00 e8 43 15 f9 ff 48 be 02 00 00 00 0a 00 00 00 48 89 30 48 8b 4d 00 8b 55 08 48 c7 40 18 00 00 00 00 48 89 48 08 89 50 10 <48> 89 45 00 c7 45 08 0a 01 00 00 48 89 03 c7 43 08 0a 01 00 00 e9
 php-fpm[169427]: segfault at 7fe80e24e0a8 ip 0000559fa9214858 sp 00007ffdf392db20 error 4 in php-fpm[559fa8f3c000+3f3000]
 Code: f8 10 0f 84 aa 01 00 00 49 8b 46 18 f6 40 50 01 0f 85 94 00 00 00 49 8b 46 18 48 8b 40 50 48 8b 00 48 85 c0 0f 84 a0 00 00 00 <8b> 10 83 fa 01 76 2c f6 40 04 40 75 05 83 ea 01 89 10 48 89 c7 e8

When this happens, matomo displays the following error:

Oops… there was a problem during the request. Maybe the server had a temporary issue, or maybe you requested a report with too much data. Please try again. If this error occurs repeatedly please contact your Matomo administrator for assistance.

Possible Solution

No idea. Since this is a production server, I can't run debug stuff.

Temporarily, I can run systemctl restart php-fpm and matomo goes back to working fine, but eventually it will reach a point when it starts segfaulting again.

Steps to Reproduce (for Bugs)

Unfortunately, I can't reliably reproduce the issue, it just happens randomly (I think).

Context

I just login and see real time visits. Tracking is done via the matomo-php-tracker (not the javascript code). I am using AlmaLinux, Apache 2.4.37 and PHP 7.4.30, all official packages as supported by RHEL. The server also runs other PHP projects, only Matomo is affected, no other php-fpm process has issues.

Thank you.

Your Environment

bx80 commented 1 year ago

Hi @LauraTaylorUK, thanks for reporting this.

It's difficult to say what could be causing these random segfaults. It sounds like it could a bug in PHP-FPM which is being triggered by something Matomo is doing.

If you have the PHP-FPM worker output captured then it could be useful to see if there are any PHP warning or errors being logged at the same time as the segfaults are occurring.

If you are using opcache then, if possible, it might be worth temporarily disabling it to see if that makes any difference.

If it is an option, then upgrading to PHP 8.x may also resolve the issue as PHP 7.4 just went end of life.

ghost commented 1 year ago

You mentioned opcache, so I went and deleted the tmp storage that holds opcache, restarted php-fpm and the problem is gone.

I'm guessing that the updating of matomo over the past few versions 4.13.0 -> 4.13.1 -> 4.13.2 -> 4.13.3 somehow caused the opcache to go stale.

Thank you for your help.

MatomoForumNotifications commented 11 months ago

This issue has been mentioned on Matomo forums. There might be relevant details there:

https://forum.matomo.org/t/marketplace-an-error-occurred/15818/4