sonata-project / SonataMediaBundle

Symfony SonataMediaBundle
https://docs.sonata-project.org/projects/SonataMediaBundle
MIT License
449 stars 496 forks source link

Memory leaks when creating medias #1742

Closed lukepass closed 3 years ago

lukepass commented 4 years ago

Environment

Symfony 3.4 website and console command

Sonata packages

$ composer show --latest 'sonata-project/*'
composer show --latest 'sonata-project/*'
sonata-project/admin-bundle              3.57.0 3.66.0 The missing Symfony Admin Generator
sonata-project/block-bundle              3.18.4 3.18.5 Symfony SonataBlockBundle
sonata-project/cache                     2.0.1  2.0.1  Cache library
sonata-project/core-bundle               3.17.2 3.18.0 Symfony SonataCoreBundle (abandoned)
sonata-project/datagrid-bundle           2.5.0  3.2.0  Symfony SonataDatagridBundle
sonata-project/doctrine-extensions       1.6.0  1.6.0  Doctrine2 behavioral extensions
sonata-project/doctrine-orm-admin-bundle 3.13.0 3.17.1 Symfony Sonata / Integrate Doctrine ORM into the S...
sonata-project/easy-extends-bundle       2.5.0  2.5.0  Symfony SonataEasyExtendsBundle
sonata-project/exporter                  2.2.0  2.2.0  Lightweight Exporter library
sonata-project/formatter-bundle          4.1.3  4.1.3  Symfony SonataFormatterBundle
sonata-project/media-bundle              3.23.1 3.24.0 Symfony SonataMediaBundle
sonata-project/translation-bundle        2.5.0  2.5.0  SonataTranslationBundle
sonata-project/user-bundle               4.5.1  4.5.3  Symfony SonataUserBundle

Symfony packages

$ composer show --latest 'symfony/*'
symfony/http-client           v5.0.8  v5.0.8  Symfony HttpClient component
symfony/http-client-contracts v2.0.1  v2.0.1  Generic abstractions related to HTTP clients
symfony/mime                  v5.0.8  v5.0.8  A library to manipulate MIME messages
symfony/monolog-bundle        v3.5.0  v3.5.0  Symfony MonologBundle
symfony/phpunit-bridge        v5.0.8  v5.0.8  Symfony PHPUnit Bridge
symfony/polyfill-apcu         v1.15.0 v1.16.0 Symfony polyfill backporting apcu_* functions to lower PHP ...
symfony/polyfill-ctype        v1.15.0 v1.16.0 Symfony polyfill for ctype functions
symfony/polyfill-intl-icu     v1.15.0 v1.16.0 Symfony polyfill for intl's ICU-related data and classes
symfony/polyfill-intl-idn     v1.15.0 v1.16.0 Symfony polyfill for intl's idn_to_ascii and idn_to_utf8 fu...
symfony/polyfill-mbstring     v1.15.0 v1.16.0 Symfony polyfill for the Mbstring extension
symfony/polyfill-php56        v1.15.0 v1.16.0 Symfony polyfill backporting some PHP 5.6+ features to lowe...
symfony/polyfill-php70        v1.15.0 v1.16.0 Symfony polyfill backporting some PHP 7.0+ features to lowe...
symfony/polyfill-php72        v1.15.0 v1.16.0 Symfony polyfill backporting some PHP 7.2+ features to lowe...
symfony/polyfill-php73        v1.15.0 v1.16.0 Symfony polyfill backporting some PHP 7.3+ features to lowe...
symfony/polyfill-util         v1.15.0 v1.16.0 Symfony utilities for portability of PHP codes
symfony/security-acl          v3.0.4  v3.0.4  Symfony Security Component - ACL (Access Control List)
symfony/service-contracts     v2.0.1  v2.0.1  Generic abstractions related to writing services
symfony/swiftmailer-bundle    v2.6.7  v3.4.0  Symfony SwiftmailerBundle
symfony/symfony               v3.4.40 v5.0.8  The Symfony PHP framework

PHP version

$ php -v
PHP 7.2.24-0ubuntu0.18.04.4 (cli) (built: Apr  8 2020 15:45:57) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.2.24-0ubuntu0.18.04.4, Copyright (c) 1999-2018, by Zend Technologies

Subject

Hello, I have a long-running command which imports a list of products (about 50,000) and each product can have more than one image.

There are a lot of media creations using this code:

public function createMedia($file, $name, $providerName, $context = 'default')
{
    $media = new Media();

    $media->setName($name);
    $media->setBinaryContent($file);
    $media->setContext($context);
    $media->setProviderName($providerName);

    // image sizes
    if ('sonata.media.provider.image' === $providerName && false !== ($sizes = @getimagesize($file))) {
        $media->setWidth($sizes[0]);
        $media->setHeight($sizes[1]);
    }

    return $media;
}

I noticed that after a while my command starts consuming a lot of memory. I am using a lot of best practices when importing products, taken directly from the Doctrine website:

if ($flush || 0 === ($flushIndex % $batchSize)) {
    // flush
    $this->em->flush();
    $this->em->clear();

    gc_collect_cycles();
    usleep($sleep);
}

If I comment the persist of media, the used memory stays stable and I don't have any leaks.

Using a profiler (I'm not so sure about this), it seems that the culprit are the postPersist events generated by the media creation, probably the resizer configured in the config.yml are consuming a lot of memory and leaving leaks somewhere?

Is there something I can do about this?

This is an image taken from the profiler:

image

Thanks!

jorrit commented 3 years ago

I think this is caused because Gaufrette's filesystem keeps every file it opens in a registry. Also, whenever contents are written these contents are saved in memory.

I solved this by calling $this->getFilesystem()->clearFileRegister(); in an override of FileProvider->setFileContents(). Perhaps Sonata Media Bundle can call $this->getFilesystem()->removeFromRegister(); in FileProvider->setFileContents().

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

dali-rajab commented 3 years ago

thank you @jorrit , you saved my day, in fact you inspired me to get my solution without overriding anything, i just get the "sonata.media.provider.image" by Dependency Injection, and i use it like this :

<?php

namespace App\Command\ACME;

use Sonata\MediaBundle\Command\BaseCommand;
use Sonata\MediaBundle\Provider\ImageProvider;

class MyCommand extends BaseCommand
{
    private $sonataImageProvider;

    public function __construct(ImageProvider $sonataImageProvider)
    {
        parent::__construct();

        $this->sonataImageProvider = $sonataImageProvider;
    }

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        foreach ($variable as $key => $value) {
            // ...some business logic ...
            $this->entityManager->persist($someEntity);
            $this->entityManager->flush();
            // and here is the solution
            $this->sonataImageProvider->getFilesystem()->clearFileRegister();
        }

    }
}