[BUG] client object increasing in size over multiple requests

[manuelKruesi]

What is the bug?

When performing multiple bulk imports with the same client object, the size of the client object will increase each time

How can one reproduce the bug?

build a String that can be ingested in the bulkimport function. Now make a loop and import this string multiple times with the same client object and log the memory size used each time, you will see an increase.

What is the expected behavior?

I expected the client Object to stay on mostly the same memory usage even over multiple requests

What is your host/environment?

PHP 8.1.27 Build System: Linux

Do you have any additional context?

I am using only one client object to prevent our host provider from blocking me as a possible ddos attacker. Therefore i made the client object on __contruct in the import class:

$this->client = \OpenSearch\ClientBuilder::fromConfig(array(
            'hosts' => array(
                $this->config['protocol'] . $host . ":" . $this->config['port']
            ),
            'retries' => 2,
            'handler' => \OpenSearch\ClientBuilder::multiHandler()
        ));

Now i loop over all the Data that needs to be imported (A lot) but after a while, the importer crashes - exhausted memory. I am paginating over all the results in steps of 500, thats why theres so many bulkimports after one another. The crash happens after about 150 bulkimport calls with 1GB of allowed memory size.

If I am misusing the client object and it is not a bug I would be happy to get some hints for how to do it better. Cheers

[shyim]

Do you configured any logger? Is the logger maybe never flushing and having everything in the RAM?

[manuelKruesi]

see the attached client construction, no logger is defined. Should I define one or can i disable logging?

[plumthedev]

I had the same issue and removing logger solved the issue.

[dblock]

@plumthedev Is it a bug in the client? Help us fix it, or document how to avoid it?

[plumthedev]

I don't think so, that is a bug in client, but I can check it in a day.

[shyim]

Do you use monolog? Did you configured a buffer_size? Otherwise it keeps everything in RAM and flushes everything at the end

[plumthedev]

@shyim I was using Laravel logger configured on config level like this:

'opensearch' => [
    'days'           => 7,
    'driver'         => 'daily',
    'formatter'   => JsonFormatter::class,
    'level'           => 'debug',
    'path'           => storage_path('logs/opensearch.jsonl'),
],

Which is using Monolog RotatingFileHandler

I don't see any option to set the buffer_size here, even in the source code of Monolog

[plumthedev]

I opened a discussion on Laravel community page: https://github.com/laravel/framework/discussions/52659 Maybe there is a solution 👀