search

danog / MadelineProto

Async PHP client API for the telegram MTProto protocol
https://docs.madelineproto.xyz
GNU Affero General Public License v3.0
2.74k stars 617 forks source link

Memory leaks in handling updates loop #1485

Closed vtsykun closed 4 months ago

vtsykun commented 4 months ago

I noticed large memory leaks when processing a large number of messages. After investigation I found out that these are related to Peer, Username, FileReference database. So I created a special a Least Recently Used (LRU) MemoryArray DbArray adapter with limited size. This partially solved the problem, memory usage reduced from +500 MB to 60 MB after restart, but there are still memory leaks

2024-02-17 06:41:10.849338 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 62.64 MB, PHP Real 44 MB, now 40.14 MB, peak 50 MB
2024-02-17 06:56:36.449571 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 67.48 MB, PHP Real 48 MB, now 35.96 MB, peak 52 MB
2024-02-17 07:12:02.050498 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 66.29 MB, PHP Real 46 MB, now 37.74 MB, peak 52 MB
2024-02-17 07:27:27.651332 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 66.34 MB, PHP Real 46 MB, now 40.2 MB, peak 54 MB
2024-02-17 07:42:53.252431 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 68.43 MB, PHP Real 48 MB, now 42.51 MB, peak 56 MB
2024-02-17 07:58:18.852877 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 70.45 MB, PHP Real 50 MB, now 44.72 MB, peak 56 MB
2024-02-17 08:13:44.454213 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 72.06 MB, PHP Real 52.5 MB, now 47.78 MB, peak 60.5 MB
2024-02-17 08:29:10.055421 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 74.29 MB, PHP Real 54.5 MB, now 50.68 MB, peak 62.5 MB
2024-02-17 08:44:35.656697 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 76.52 MB, PHP Real 56.5 MB, now 53.29 MB, peak 64.5 MB
2024-02-17 09:00:01.256927 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 78.92 MB, PHP Real 58.5 MB, now 50.1 MB, peak 66.5 MB
2024-02-17 09:15:26.856792 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 81.16 MB, PHP Real 60.5 MB, now 53.36 MB, peak 66.5 MB
2024-02-17 09:30:52.456841 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 83.9 MB, PHP Real 65 MB, now 58.55 MB, peak 73 MB
2024-02-17 09:46:18.057578 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 86.16 MB, PHP Real 67 MB, now 61.33 MB, peak 75 MB
2024-02-17 10:01:43.658044 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 88.35 MB, PHP Real 69 MB, now 63.51 MB, peak 77 MB
2024-02-17 10:17:09.258085 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 90.57 MB, PHP Real 71 MB, now 65.88 MB, peak 79 MB
2024-02-17 10:32:34.858481 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 92.77 MB, PHP Real 73 MB, now 68.2 MB, peak 81 MB
2024-02-17 10:48:00.459272 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 95 MB, PHP Real 75 MB, now 70.74 MB, peak 83 MB
2024-02-17 11:03:26.060398 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 97.05 MB, PHP Real 77 MB, now 67.43 MB, peak 83 MB
2024-02-17 11:18:51.661908 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 97.74 MB, PHP Real 79 MB, now 70.55 MB, peak 85 MB
2024-02-17 11:34:17.262058 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 101.57 MB, PHP Real 81 MB, now 73.18 MB, peak 87 MB
2024-02-17 11:49:42.863397 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 104.82 MB, PHP Real 88 MB, now 80.95 MB, peak 96 MB
2024-02-17 12:05:08.463539 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 107.03 MB, PHP Real 90 MB, now 83.23 MB, peak 96 MB
2024-02-17 12:20:34.063989 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 109.24 MB, PHP Real 92 MB, now 85.53 MB, peak 98 MB
2024-02-17 12:35:59.664383 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 111.51 MB, PHP Real 94 MB, now 88.59 MB, peak 102 MB
2024-02-17 12:51:25.265648 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 113.62 MB, PHP Real 96 MB, now 84.8 MB, peak 102 MB
2024-02-17 13:06:50.865821 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 113.83 MB, PHP Real 96 MB, now 86.91 MB, peak 102 MB
2024-02-17 13:22:16.466524 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 116.01 MB, PHP Real 98 MB, now 89.1 MB, peak 104 MB
2024-02-17 13:37:42.066671 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 118.12 MB, PHP Real 100 MB, now 91.43 MB, peak 106 MB
2024-02-17 13:53:07.666765 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 118.37 MB, PHP Real 100 MB, now 93.23 MB, peak 108 MB
2024-02-17 14:08:33.267200 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 120.57 MB, PHP Real 102 MB, now 95.54 MB, peak 110 MB
2024-02-17 14:23:58.866784 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 122.72 MB, PHP Real 104 MB, now 97.44 MB, peak 112 MB
2024-02-17 14:39:24.467094 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 124.93 MB, PHP Real 106 MB, now 93.29 MB, peak 112 MB
2024-02-17 14:54:50.066821 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 125.07 MB, PHP Real 106 MB, now 95.34 MB, peak 112 MB
2024-02-17 15:10:15.666966 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 125.36 MB, PHP Real 106 MB, now 97.39 MB, peak 114 MB
2024-02-17 15:25:41.267423 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 127.5 MB, PHP Real 108 MB, now 99.06 MB, peak 114 MB
2024-02-17 15:41:06.868354 docker-entrypoint, xxxxxxxxx:    Memory usage check: RSS 129.65 MB, PHP Real 110 MB, now 100.97 MB, peak 118 MB

Then I tried to debug with https://github.com/BitOne/php-meminfo, but seems this library not dump Fibers memory stack dump

\Revolt\EventLoop::onSignal(\SIGUSR1, function () {
    echo "\n---------\n\n\n-SIGUSR1-\n\n\n";
    $file = is_writable("/data") ? '/data/mem_dump.json' : '/tmp/mem_dump.json';
    if (\file_exists($file)) {
        @\unlink($file);
    }
    \meminfo_dump(\fopen($file, 'w'));
});

What places in the code can I check, (for example where local caching is used) to fix memory leaks?

Teneja commented 4 months ago

Roberto Bussolari Il giorno 18 feb 2024, alle ore 13:25, Uladzimir Tsykun @.***> ha scritto: I noticed large memory leaks when processing a large number of messages. After investigation I found out that these are related to Peer, Username, FileReference database. So I created a special a Least Recently Used (LRU) MemoryArray DbArray adapter with limited size. This partially solved the problem, memory usage reduced from +500 MB to 60 MB after restart, but there are still memory leaks 2024-02-17 06:41:10.849338 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 62.64 MB, PHP Real 44 MB, now 40.14 MB, peak 50 MB 2024-02-17 06:56:36.449571 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 67.48 MB, PHP Real 48 MB, now 35.96 MB, peak 52 MB 2024-02-17 07:12:02.050498 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 66.29 MB, PHP Real 46 MB, now 37.74 MB, peak 52 MB 2024-02-17 07:27:27.651332 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 66.34 MB, PHP Real 46 MB, now 40.2 MB, peak 54 MB 2024-02-17 07:42:53.252431 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 68.43 MB, PHP Real 48 MB, now 42.51 MB, peak 56 MB 2024-02-17 07:58:18.852877 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 70.45 MB, PHP Real 50 MB, now 44.72 MB, peak 56 MB 2024-02-17 08:13:44.454213 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 72.06 MB, PHP Real 52.5 MB, now 47.78 MB, peak 60.5 MB 2024-02-17 08:29:10.055421 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 74.29 MB, PHP Real 54.5 MB, now 50.68 MB, peak 62.5 MB 2024-02-17 08:44:35.656697 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 76.52 MB, PHP Real 56.5 MB, now 53.29 MB, peak 64.5 MB 2024-02-17 09:00:01.256927 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 78.92 MB, PHP Real 58.5 MB, now 50.1 MB, peak 66.5 MB 2024-02-17 09:15:26.856792 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 81.16 MB, PHP Real 60.5 MB, now 53.36 MB, peak 66.5 MB 2024-02-17 09:30:52.456841 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 83.9 MB, PHP Real 65 MB, now 58.55 MB, peak 73 MB 2024-02-17 09:46:18.057578 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 86.16 MB, PHP Real 67 MB, now 61.33 MB, peak 75 MB 2024-02-17 10:01:43.658044 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 88.35 MB, PHP Real 69 MB, now 63.51 MB, peak 77 MB 2024-02-17 10:17:09.258085 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 90.57 MB, PHP Real 71 MB, now 65.88 MB, peak 79 MB 2024-02-17 10:32:34.858481 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 92.77 MB, PHP Real 73 MB, now 68.2 MB, peak 81 MB 2024-02-17 10:48:00.459272 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 95 MB, PHP Real 75 MB, now 70.74 MB, peak 83 MB 2024-02-17 11:03:26.060398 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 97.05 MB, PHP Real 77 MB, now 67.43 MB, peak 83 MB 2024-02-17 11:18:51.661908 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 97.74 MB, PHP Real 79 MB, now 70.55 MB, peak 85 MB 2024-02-17 11:34:17.262058 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 101.57 MB, PHP Real 81 MB, now 73.18 MB, peak 87 MB 2024-02-17 11:49:42.863397 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 104.82 MB, PHP Real 88 MB, now 80.95 MB, peak 96 MB 2024-02-17 12:05:08.463539 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 107.03 MB, PHP Real 90 MB, now 83.23 MB, peak 96 MB 2024-02-17 12:20:34.063989 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 109.24 MB, PHP Real 92 MB, now 85.53 MB, peak 98 MB 2024-02-17 12:35:59.664383 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 111.51 MB, PHP Real 94 MB, now 88.59 MB, peak 102 MB 2024-02-17 12:51:25.265648 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 113.62 MB, PHP Real 96 MB, now 84.8 MB, peak 102 MB 2024-02-17 13:06:50.865821 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 113.83 MB, PHP Real 96 MB, now 86.91 MB, peak 102 MB 2024-02-17 13:22:16.466524 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 116.01 MB, PHP Real 98 MB, now 89.1 MB, peak 104 MB 2024-02-17 13:37:42.066671 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 118.12 MB, PHP Real 100 MB, now 91.43 MB, peak 106 MB 2024-02-17 13:53:07.666765 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 118.37 MB, PHP Real 100 MB, now 93.23 MB, peak 108 MB 2024-02-17 14:08:33.267200 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 120.57 MB, PHP Real 102 MB, now 95.54 MB, peak 110 MB 2024-02-17 14:23:58.866784 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 122.72 MB, PHP Real 104 MB, now 97.44 MB, peak 112 MB 2024-02-17 14:39:24.467094 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 124.93 MB, PHP Real 106 MB, now 93.29 MB, peak 112 MB 2024-02-17 14:54:50.066821 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 125.07 MB, PHP Real 106 MB, now 95.34 MB, peak 112 MB 2024-02-17 15:10:15.666966 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 125.36 MB, PHP Real 106 MB, now 97.39 MB, peak 114 MB 2024-02-17 15:25:41.267423 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 127.5 MB, PHP Real 108 MB, now 99.06 MB, peak 114 MB 2024-02-17 15:41:06.868354 docker-entrypoint, xxxxxxxxx: Memory usage check: RSS 129.65 MB, PHP Real 110 MB, now 100.97 MB, peak 118 MB

Then I tried to debug with https://github.com/BitOne/php-meminfo, but seems this library not dump Fibers memory stack dump \Revolt\EventLoop::onSignal(\SIGUSR1, function () { echo "\n---------\n\n\n-SIGUSR1-\n\n\n"; $file = is_writable("/data") ? '/data/mem_dump.json' : '/tmp/mem_dump.json'; if (\file_exists($file)) { @\unlink($file); } \meminfo_dump(\fopen($file, 'w')); }); What places in the code can I check, (for example where local caching is used) to fix memory leaks?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>

danog commented 4 months ago

You're using the event handler, and the latest v8 beta, correct? The behavior of the DbArray is not a memory leak, it is the intended behavior, caching database entries into RAM for the period of time specified in the settings.

This behavior can be turned off altogether by specifying a TTL of 0 in the settings, at the expense of performance, which is something I do not recommend doing.

vtsykun commented 4 months ago

Hi @danog I found it. Problem in MinDatabase cache

final class MinDatabase implements TLCallback
{
    /**
     * Temporary cache during deserialization.
     */
    private array $cache = [];

I use my own fork for debug memory https://github.com/vtsykun/php-meminfo where fixed bugs https://github.com/BitOne/php-meminfo/issues/133 https://github.com/BitOne/php-meminfo/issues/135

$ php bin/analyzer top-children tmp/mem_dump.json0
+-----+----------------+----------+
| Num | Item ids       | Children |
+-----+----------------+----------+
| 1   | 0x7fd2d6099a48 | 120357   |
| 2   | 0x7fd2d3cdce80 | 6391     |
| 3   | 0x7fd2d604e8e8 | 1472     |
| 4   | 0x7fd2d6d48c98 | 1301     |
| 5   | 0x7fd2d6d48c88 | 1195     |
+-----+----------------+----------+

Path from 0x7fd2dad5ae00
+------------------------+
| Id: 0x7fd2d6099a48     |
| Type: array            |
| Size: 72 B             |
| Is root: No            |
| Children count: 120357 |
+------------------------+
         ^          
         |          
       cache        
         |          
         |          
+-----------------------------------------------------+
| Id: 0x7fd2d7fc7820                                  |
| Type: object                                        |
| Class: danog\MadelineProto\MTProtoTools\MinDatabase |
| Object Handle: 164                                  |
| Size: 72 B                                          |
| Is root: No                                         |
| Children count: 6                                   |
+-----------------------------------------------------+
         ^          
         |          
    minDatabase     
         |          
         |          
+------------------------------------+
| Id: 0x7fd2d3de9800                 |
| Type: object                       |
| Class: danog\MadelineProto\MTProto |
| Object Handle: 159                 |
| Size: 72 B                         |
| Is root: No                        |
| Children count: 78                 |
+------------------------------------+
danog commented 4 months ago

Thank you for investigating and finding the issue, fixed in the latest commit!

vtsykun commented 4 months ago

Hi @danog it's works now.

Thank you for the fast fix and nice software. 

2024-02-20 11:55:50.785497  Memory usage: RSS 72.50 MB, PHP Heap 50 MB, mean 40.26 MB, peak 56 MB
2024-02-20 12:11:16.433019  Memory usage: RSS 75.43 MB, PHP Heap 54 MB, mean 43.60 MB, peak 56 MB
2024-02-20 12:26:42.033885  Memory usage: RSS 72.62 MB, PHP Heap 50 MB, mean 41.31 MB, peak 56 MB
2024-02-20 12:42:07.633855  Memory usage: RSS 72.64 MB, PHP Heap 50 MB, mean 42.71 MB, peak 56 MB
2024-02-20 12:57:33.233858  Memory usage: RSS 72.62 MB, PHP Heap 50 MB, mean 43.31 MB, peak 56 MB
2024-02-20 13:12:58.834414  Memory usage: RSS 72.63 MB, PHP Heap 50 MB, mean 43.73 MB, peak 56 MB
2024-02-20 13:28:24.434696  Memory usage: RSS 72.63 MB, PHP Heap 50 MB, mean 44.66 MB, peak 56 MB
2024-02-20 13:43:50.034722  Memory usage: RSS 72.64 MB, PHP Heap 50 MB, mean 45.24 MB, peak 56 MB
2024-02-20 13:59:15.635520  Memory usage: RSS 72.63 MB, PHP Heap 50 MB, mean 38.67 MB, peak 56 MB
2024-02-20 14:14:41.236329  Memory usage: RSS 72.63 MB, PHP Heap 50 MB, mean 39.54 MB, peak 56 MB
2024-02-20 14:30:06.836919  Memory usage: RSS 72.63 MB, PHP Heap 50 MB, mean 40.18 MB, peak 56 MB
2024-02-20 14:45:32.436505  Memory usage: RSS 72.64 MB, PHP Heap 50 MB, mean 40.60 MB, peak 56 MB