search

reactphp / http

Event-driven, streaming HTTP client and server implementation for ReactPHP.
https://reactphp.org/http/
MIT License
744 stars 143 forks source link

Memory leak when using persistent connections #514

Closed mmoreram closed 7 months ago

mmoreram commented 9 months ago

When using persistent connections, this code makes me think that could cause problems... This happens inside the handleRequest in StreamingServer file. If we use persistent connections, we are adding one close event per request, causing all responses being referenced, even when it is returned properly (not allowing GC make its work).

// cancel pending promise once connection closes
if ($response instanceof PromiseInterface && \method_exists($response, 'cancel')) {
    $conn->on('close', function () use ($response) {
        $response->cancel();
    });
}
WyriHaximus commented 8 months ago

Hey @mmoreram, just tried to see if keep alive indeed leaks memory due to that. And I can't prove that it does. Modified example 51 into this:

<?php

use React\EventLoop\Loop;

$reportMemory = static function () {
    $rawInt = memory_get_usage(false);
    $rawExt = memory_get_usage(true);
    echo date('r'), ' Internal: ', ($rawInt / 1024), 'KB External: ', ($rawExt / 1024), 'KB (Raw Internal: ', $rawInt, ' Raw External: ', $rawExt, ')', PHP_EOL;
};

$reportMemory();
echo 'Loading autoloader', PHP_EOL;
require __DIR__ . '/../vendor/autoload.php';

$reportMemory();
echo 'Creating HTTP Server', PHP_EOL;
$http = new React\Http\HttpServer(function (Psr\Http\Message\ServerRequestInterface $request) {
    return React\Http\Message\Response::plaintext(
        "Hello World!\n"
    );
});

$reportMemory();
echo 'Creating Socket Server', PHP_EOL;
$socket = new React\Socket\SocketServer(isset($argv[1]) ? $argv[1] : '0.0.0.0:0');

$reportMemory();
echo 'Listening on socket', PHP_EOL;
$http->listen($socket);

Loop::addPeriodicTimer(60, $reportMemory);

echo 'Listening on ' . str_replace('tcp:', 'http:', $socket->getAddress()) . PHP_EOL;

$reportMemory();
echo 'Starting Event Loop', PHP_EOL;

And then ran ab against it with the following arguments to do 100.000.000 requests against it with 100 keep alive connections:

ab -k -c 100 -n 100000000 http://localhost:42353/

It came back after just over an hour with this:

image

This is the output from the server. As you can see it stays stable at 3.9MB of memory used by PHP and 6MB allocated with the system throughout the test. After completed it dropped back to just over 2MB. I would love to have more information on what makes this a memory leak for you.

Sat, 17 Feb 2024 18:29:57 +0100 Internal: 402.5546875KB External: 2048KB (Raw Internal: 412216 Raw External: 2097152)                                                                                      
Loading autoloader                                                                                                                                                                                         
Sat, 17 Feb 2024 18:29:57 +0100 Internal: 1262.78125KB External: 2048KB (Raw Internal: 1293088 Raw External: 2097152)                                                                                      
Creating HTTP Server                                                                                                                                                                                       
Sat, 17 Feb 2024 18:29:57 +0100 Internal: 1599.9921875KB External: 2048KB (Raw Internal: 1638392 Raw External: 2097152)                                                                                    
Creating Socket Server                                                                                                                                                                                     
Sat, 17 Feb 2024 18:29:57 +0100 Internal: 1658.4453125KB External: 2048KB (Raw Internal: 1698248 Raw External: 2097152)                                                                                    
Listening on socket                                                                                                                                                                                        
Listening on http://0.0.0.0:42353                                                                                                                                                                          
Sat, 17 Feb 2024 18:29:57 +0100 Internal: 1667.5859375KB External: 2048KB (Raw Internal: 1707608 Raw External: 2097152)                                                                                    
Starting Event Loop                                                                                                                                                                                        
Sat, 17 Feb 2024 18:30:57 +0100 Internal: 3951.8046875KB External: 6144KB (Raw Internal: 4046648 Raw External: 6291456)                                                                                    
Sat, 17 Feb 2024 18:31:57 +0100 Internal: 3938.9140625KB External: 6144KB (Raw Internal: 4033448 Raw External: 6291456)                                                                                    
Sat, 17 Feb 2024 18:32:57 +0100 Internal: 3935.984375KB External: 6144KB (Raw Internal: 4030448 Raw External: 6291456)                                                                                     
Sat, 17 Feb 2024 18:33:57 +0100 Internal: 3947.1171875KB External: 6144KB (Raw Internal: 4041848 Raw External: 6291456)                                                                                    
Sat, 17 Feb 2024 18:34:57 +0100 Internal: 3955.90625KB External: 6144KB (Raw Internal: 4050848 Raw External: 6291456)                                                                                      
Sat, 17 Feb 2024 18:35:57 +0100 Internal: 3964.109375KB External: 6144KB (Raw Internal: 4059248 Raw External: 6291456)                                                                                     
Sat, 17 Feb 2024 18:36:57 +0100 Internal: 3961.1796875KB External: 6144KB (Raw Internal: 4056248 Raw External: 6291456)                                                                                    
Sat, 17 Feb 2024 18:37:57 +0100 Internal: 3940.671875KB External: 6144KB (Raw Internal: 4035248 Raw External: 6291456)                                                                                     
Sat, 17 Feb 2024 18:38:57 +0100 Internal: 3958.8359375KB External: 6144KB (Raw Internal: 4053848 Raw External: 6291456)                                                                                    
Sat, 17 Feb 2024 18:39:57 +0100 Internal: 3962.9375KB External: 6144KB (Raw Internal: 4058048 Raw External: 6291456)                                                                                       
Sat, 17 Feb 2024 18:40:57 +0100 Internal: 3951.8046875KB External: 6144KB (Raw Internal: 4046648 Raw External: 6291456)                                                                                    
Sat, 17 Feb 2024 18:41:57 +0100 Internal: 3951.8046875KB External: 6144KB (Raw Internal: 4046648 Raw External: 6291456)                                                                                    
Sat, 17 Feb 2024 18:42:57 +0100 Internal: 3938.9140625KB External: 6144KB (Raw Internal: 4033448 Raw External: 6291456)                                                                                    
Sat, 17 Feb 2024 18:43:57 +0100 Internal: 3956.4921875KB External: 6144KB (Raw Internal: 4051448 Raw External: 6291456)                                                                                    
Sat, 17 Feb 2024 18:44:57 +0100 Internal: 3967.0390625KB External: 6144KB (Raw Internal: 4062248 Raw External: 6291456)                                                                                    
Sat, 17 Feb 2024 18:45:57 +0100 Internal: 3954.1484375KB External: 6144KB (Raw Internal: 4049048 Raw External: 6291456)                                                                                    
Sat, 17 Feb 2024 18:46:57 +0100 Internal: 3942.4296875KB External: 6144KB (Raw Internal: 4037048 Raw External: 6291456)                                                                                    
Sat, 17 Feb 2024 18:47:57 +0100 Internal: 3959.421875KB External: 6144KB (Raw Internal: 4054448 Raw External: 6291456)                                                                                     
Sat, 17 Feb 2024 18:48:57 +0100 Internal: 3941.84375KB External: 6144KB (Raw Internal: 4036448 Raw External: 6291456)                                                                                      
Sat, 17 Feb 2024 18:49:57 +0100 Internal: 3955.90625KB External: 6144KB (Raw Internal: 4050848 Raw External: 6291456)                                                                                      
Sat, 17 Feb 2024 18:50:57 +0100 Internal: 3945.359375KB External: 6144KB (Raw Internal: 4040048 Raw External: 6291456)                                                                                     
Sat, 17 Feb 2024 18:51:57 +0100 Internal: 3928.953125KB External: 6144KB (Raw Internal: 4023248 Raw External: 6291456)                                                                                     
Sat, 17 Feb 2024 18:52:57 +0100 Internal: 3923.6796875KB External: 6144KB (Raw Internal: 4017848 Raw External: 6291456)                                                                                    
Sat, 17 Feb 2024 18:53:57 +0100 Internal: 3932.46875KB External: 6144KB (Raw Internal: 4026848 Raw External: 6291456)                                                                                      
Sat, 17 Feb 2024 18:54:57 +0100 Internal: 3937.7421875KB External: 6144KB (Raw Internal: 4032248 Raw External: 6291456)                                                                                    
Sat, 17 Feb 2024 18:55:57 +0100 Internal: 3966.453125KB External: 6144KB (Raw Internal: 4061648 Raw External: 6291456)                                                                                                      
Sat, 17 Feb 2024 18:56:57 +0100 Internal: 3955.90625KB External: 6144KB (Raw Internal: 4050848 Raw External: 6291456)
Sat, 17 Feb 2024 18:57:57 +0100 Internal: 3945.359375KB External: 6144KB (Raw Internal: 4040048 Raw External: 6291456)
Sat, 17 Feb 2024 18:58:57 +0100 Internal: 3942.4296875KB External: 6144KB (Raw Internal: 4037048 Raw External: 6291456)
Sat, 17 Feb 2024 18:59:57 +0100 Internal: 3961.765625KB External: 6144KB (Raw Internal: 4056848 Raw External: 6291456)
Sat, 17 Feb 2024 19:00:57 +0100 Internal: 3954.1484375KB External: 6144KB (Raw Internal: 4049048 Raw External: 6291456)
Sat, 17 Feb 2024 19:01:57 +0100 Internal: 3923.6796875KB External: 6144KB (Raw Internal: 4017848 Raw External: 6291456)
Sat, 17 Feb 2024 19:02:57 +0100 Internal: 3954.734375KB External: 6144KB (Raw Internal: 4049648 Raw External: 6291456)
Sat, 17 Feb 2024 19:03:57 +0100 Internal: 3934.8125KB External: 6144KB (Raw Internal: 4029248 Raw External: 6291456)
Sat, 17 Feb 2024 19:04:57 +0100 Internal: 3959.421875KB External: 6144KB (Raw Internal: 4054448 Raw External: 6291456)
Sat, 17 Feb 2024 19:05:57 +0100 Internal: 3964.109375KB External: 6144KB (Raw Internal: 4059248 Raw External: 6291456)
Sat, 17 Feb 2024 19:06:57 +0100 Internal: 3928.953125KB External: 6144KB (Raw Internal: 4023248 Raw External: 6291456)
Sat, 17 Feb 2024 19:07:57 +0100 Internal: 3956.4921875KB External: 6144KB (Raw Internal: 4051448 Raw External: 6291456)
Sat, 17 Feb 2024 19:08:57 +0100 Internal: 3964.6953125KB External: 6144KB (Raw Internal: 4059848 Raw External: 6291456)
Sat, 17 Feb 2024 19:09:57 +0100 Internal: 3929.5390625KB External: 6144KB (Raw Internal: 4023848 Raw External: 6291456)
Sat, 17 Feb 2024 19:10:57 +0100 Internal: 3950.6328125KB External: 6144KB (Raw Internal: 4045448 Raw External: 6291456)
Sat, 17 Feb 2024 19:11:57 +0100 Internal: 3962.9375KB External: 6144KB (Raw Internal: 4058048 Raw External: 6291456)
Sat, 17 Feb 2024 19:12:57 +0100 Internal: 3945.9453125KB External: 6144KB (Raw Internal: 4040648 Raw External: 6291456)
Sat, 17 Feb 2024 19:13:57 +0100 Internal: 3941.84375KB External: 6144KB (Raw Internal: 4036448 Raw External: 6291456)
Sat, 17 Feb 2024 19:14:57 +0100 Internal: 3937.7421875KB External: 6144KB (Raw Internal: 4032248 Raw External: 6291456)
Sat, 17 Feb 2024 19:15:57 +0100 Internal: 3947.703125KB External: 6144KB (Raw Internal: 4042448 Raw External: 6291456)
Sat, 17 Feb 2024 19:16:57 +0100 Internal: 3953.5625KB External: 6144KB (Raw Internal: 4048448 Raw External: 6291456)
Sat, 17 Feb 2024 19:17:57 +0100 Internal: 3948.875KB External: 6144KB (Raw Internal: 4043648 Raw External: 6291456)
Sat, 17 Feb 2024 19:18:57 +0100 Internal: 3936.5703125KB External: 6144KB (Raw Internal: 4031048 Raw External: 6291456)
Sat, 17 Feb 2024 19:19:57 +0100 Internal: 3947.703125KB External: 6144KB (Raw Internal: 4042448 Raw External: 6291456)
Sat, 17 Feb 2024 19:20:57 +0100 Internal: 3914.3046875KB External: 6144KB (Raw Internal: 4008248 Raw External: 6291456)
Sat, 17 Feb 2024 19:21:57 +0100 Internal: 3947.1171875KB External: 6144KB (Raw Internal: 4041848 Raw External: 6291456)
Sat, 17 Feb 2024 19:22:57 +0100 Internal: 3970.5546875KB External: 6144KB (Raw Internal: 4065848 Raw External: 6291456)
Sat, 17 Feb 2024 19:23:57 +0100 Internal: 3929.5390625KB External: 6144KB (Raw Internal: 4023848 Raw External: 6291456)
Sat, 17 Feb 2024 19:24:57 +0100 Internal: 3959.421875KB External: 6144KB (Raw Internal: 4054448 Raw External: 6291456)
Sat, 17 Feb 2024 19:25:57 +0100 Internal: 3955.3203125KB External: 6144KB (Raw Internal: 4050248 Raw External: 6291456)
Sat, 17 Feb 2024 19:26:57 +0100 Internal: 3924.265625KB External: 6144KB (Raw Internal: 4018448 Raw External: 6291456)
Sat, 17 Feb 2024 19:27:57 +0100 Internal: 3945.359375KB External: 6144KB (Raw Internal: 4040048 Raw External: 6291456)
Sat, 17 Feb 2024 19:28:57 +0100 Internal: 3936.5703125KB External: 6144KB (Raw Internal: 4031048 Raw External: 6291456)
Sat, 17 Feb 2024 19:29:57 +0100 Internal: 3939.5KB External: 6144KB (Raw Internal: 4034048 Raw External: 6291456)
Sat, 17 Feb 2024 19:30:57 +0100 Internal: 3956.4921875KB External: 6144KB (Raw Internal: 4051448 Raw External: 6291456)
Sat, 17 Feb 2024 19:31:57 +0100 Internal: 3951.21875KB External: 6144KB (Raw Internal: 4046048 Raw External: 6291456)
Sat, 17 Feb 2024 19:32:57 +0100 Internal: 3943.6015625KB External: 6144KB (Raw Internal: 4038248 Raw External: 6291456)
Sat, 17 Feb 2024 19:33:58 +0100 Internal: 3964.109375KB External: 6144KB (Raw Internal: 4059248 Raw External: 6291456)
Sat, 17 Feb 2024 19:34:58 +0100 Internal: 3926.0234375KB External: 6144KB (Raw Internal: 4020248 Raw External: 6291456)
Sat, 17 Feb 2024 19:35:58 +0100 Internal: 3935.3984375KB External: 6144KB (Raw Internal: 4029848 Raw External: 6291456)
Sat, 17 Feb 2024 19:36:57 +0100 Internal: 3939.5KB External: 6144KB (Raw Internal: 4034048 Raw External: 6291456)
Sat, 17 Feb 2024 19:37:58 +0100 Internal: 3958.25KB External: 6144KB (Raw Internal: 4053248 Raw External: 6291456)
Sat, 17 Feb 2024 19:38:58 +0100 Internal: 3944.7734375KB External: 6144KB (Raw Internal: 4039448 Raw External: 6291456)
Sat, 17 Feb 2024 19:39:58 +0100 Internal: 3944.7734375KB External: 6144KB (Raw Internal: 4039448 Raw External: 6291456)
Sat, 17 Feb 2024 19:40:58 +0100 Internal: 3954.1484375KB External: 6144KB (Raw Internal: 4049048 Raw External: 6291456)
Sat, 17 Feb 2024 19:41:58 +0100 Internal: 3950.6328125KB External: 6144KB (Raw Internal: 4045448 Raw External: 6291456)
Sat, 17 Feb 2024 19:42:58 +0100 Internal: 3936.5703125KB External: 6144KB (Raw Internal: 4031048 Raw External: 6291456)
Sat, 17 Feb 2024 19:43:58 +0100 Internal: 3911.1875KB External: 6144KB (Raw Internal: 4005056 Raw External: 6291456)
Sat, 17 Feb 2024 19:44:58 +0100 Internal: 2327.59375KB External: 6144KB (Raw Internal: 2383456 Raw External: 6291456)
Sat, 17 Feb 2024 19:45:58 +0100 Internal: 2327.59375KB External: 6144KB (Raw Internal: 2383456 Raw External: 6291456)
Sat, 17 Feb 2024 19:46:58 +0100 Internal: 2327.59375KB External: 6144KB (Raw Internal: 2383456 Raw External: 6291456)
Sat, 17 Feb 2024 19:47:58 +0100 Internal: 2327.59375KB External: 6144KB (Raw Internal: 2383456 Raw External: 6291456)
Sat, 17 Feb 2024 19:48:58 +0100 Internal: 2327.59375KB External: 6144KB (Raw Internal: 2383456 Raw External: 6291456)
Sat, 17 Feb 2024 19:49:58 +0100 Internal: 2327.59375KB External: 6144KB (Raw Internal: 2383456 Raw External: 6291456)
Sat, 17 Feb 2024 19:50:58 +0100 Internal: 2327.59375KB External: 6144KB (Raw Internal: 2383456 Raw External: 6291456)
Sat, 17 Feb 2024 19:51:58 +0100 Internal: 2327.59375KB External: 6144KB (Raw Internal: 2383456 Raw External: 6291456)
Sat, 17 Feb 2024 19:52:58 +0100 Internal: 2327.59375KB External: 6144KB (Raw Internal: 2383456 Raw External: 6291456)
Sat, 17 Feb 2024 19:53:58 +0100 Internal: 2327.59375KB External: 6144KB (Raw Internal: 2383456 Raw External: 6291456)
Sat, 17 Feb 2024 19:54:58 +0100 Internal: 2327.59375KB External: 6144KB (Raw Internal: 2383456 Raw External: 6291456)
Sat, 17 Feb 2024 19:55:58 +0100 Internal: 2327.59375KB External: 6144KB (Raw Internal: 2383456 Raw External: 6291456)
Sat, 17 Feb 2024 19:56:58 +0100 Internal: 2327.59375KB External: 6144KB (Raw Internal: 2383456 Raw External: 6291456)
Sat, 17 Feb 2024 19:57:58 +0100 Internal: 2327.59375KB External: 6144KB (Raw Internal: 2383456 Raw External: 6291456)
Sat, 17 Feb 2024 19:58:58 +0100 Internal: 2327.59375KB External: 6144KB (Raw Internal: 2383456 Raw External: 6291456)
Sat, 17 Feb 2024 19:59:58 +0100 Internal: 2327.59375KB External: 6144KB (Raw Internal: 2383456 Raw External: 6291456)
Sat, 17 Feb 2024 20:00:58 +0100 Internal: 2327.59375KB External: 6144KB (Raw Internal: 2383456 Raw External: 6291456)
Sat, 17 Feb 2024 20:01:58 +0100 Internal: 2327.59375KB External: 6144KB (Raw Internal: 2383456 Raw External: 6291456)
WyriHaximus commented 8 months ago

While having a chat with @SimonFrings this morning, we realised I wasn't using a promise in the test. After adjusting the test we confirmed the reported issue very clearly:

Mon, 19 Feb 2024 09:12:14 +0100 Internal: 1667.7109375KB External: 2048KB (Raw Internal: 1707736 Raw External: 2097152)                                                                                    
Starting Event Loop
Mon, 19 Feb 2024 09:13:14 +0100 Internal: 1638539.09375KB External: 1648640KB (Raw Internal: 1677864032 Raw External: 1688207360)

Currently working on adding unit tests to confirm the clean up of those close calls.

SimonFrings commented 7 months ago

@mmoreram Thanks for bringing this up, #515 by @WyriHaximus is now merged and should fix the reported issue. I also changed the title of this ticket to better match the described topic :+1:

If you'd like to support this development, consider sponsoring ReactPHP! ❤️ Thank you!