memory leak and errors

[gabrieleolmi]

I have a server running Ubuntu 22.04 where I have apache and php installed. I have recently started using libvips for php, and I have noticed that every time a php script is run, the amount of ram used increases more and more, without being freed. I am forced to use Vips\Config::shutDown() at the end of the script otherwise the server ram gets saturated. I tried to use Vips\Config::cacheSetMax(5) but nothing changes (memory grows more and more).

php file example:

<?php
require __DIR__ . '/vendor/autoload.php';
use Jcupitt\Vips;

$image = Vips\Image::newFromFile($inputImagePath);
//some operations on the width and height of the image
//.....
$image = Vips\Image::thumbnail($inputImagePath, $width, ['height' => $height, 'crop' => 'centre']);
$image->writeToFile($outputImagePath, ['Q' => $quality]);

Also sometimes libvips returns an error:

(process:211949): GLib-GObject-WARNING **: 16:54:47.542: cannot register existing type 'VipsObject'
(process:211949): GLib-CRITICAL **: 16:54:47.542: g_once_init_leave: assertion 'result != 0' failed
(process:211949): GLib-GObject-CRITICAL **: 16:54:47.542: g_type_register_static: assertion 'parent_
type > 0' failed
(process:211949): GLib-CRITICAL **: 16:54:47.542: g_once_init_leave: assertion 'result != 0' failed

libvips42 version: 8.12.1 php-vips version: 2.0.3

[jcupitt]

Hi @GabrieleOlmi,

How is apache set up? I suppose you have fastcgi and a persistent process for requests, is that right?

[gabrieleolmi]

FPM/FastCGI

[kleisauke]

Perhaps this is a symptom of https://www.php.net/manual/en/language.oop5.decon.php#105368? I tried to debug that using commit https://github.com/kleisauke/php-vips/commit/d20ec0d1857b27cdb30c661e38971de38c9c1afb and see about 45 references still alive after the OK print on the test suite.

[jcupitt]

Huh that's a bit surprising, there shouldn't be any circular references.

Does adding gc_collect_cycles() to your php fix this?

[kleisauke]

Ah, my bad. You're right, there shouldn't be any circular references. Commit https://github.com/kleisauke/php-vips/commit/73d05c08a3b5520d124eb2edf0f4ccb0ddbbdc4e fixes that, it seems we need to explicitly unset the images in tearDown() (just like teardown_class in libvips' test suite).

[kleisauke]

The cannot register existing type error can probably be fixed if libvips is linked with -Wl,-z,nodelete, see: https://github.com/libvips/php-vips/issues/142#issuecomment-1188960441

Hopefully that would also fix the memory leak you were seeing.

[jcupitt]

@GabrieleOlmi libvips 8.13 is out now with Kleis's linking improvement (setting nodelete).

https://github.com/libvips/libvips/releases

Any testing would be great.

[gabrieleolmi]

I tried building from source following the instructions on the official page https://www.libvips.org/install.html but ninja test returns an error:

[0/1] Running all tests.
1/8 cli                OK               1.49s
2/8 seq                FAIL             0.03s   exit status 1
>>> MALLOC_PERTURB_=105 /home/ubuntu/vips-8.13.0/test/test_seq.sh
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
stdout:
building huge test PNG image ... ok
testing vipsthumbnail ... vipsthumbnail failed in basic mode
stderr:
VipsForeignSave: "/home/ubuntu/vips-8.13.0/test/tmp-20855/huge.png" is not a known file format
/home/ubuntu/vips-8.13.0/build-dir/tools/vipsthumbnail: unable to thumbnail /home/ubuntu/vips-8.13.0/test/tmp-20855/huge.png
VipsForeignLoad: file "/home/ubuntu/vips-8.13.0/test/tmp-20855/huge.png" does not exist
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

3/8 formats            FAIL             2.16s   exit status 1
>>> MALLOC_PERTURB_=179 /home/ubuntu/vips-8.13.0/test/test_formats.sh
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
stdout:
testing sample.jpg v ... ok
testing sample.jpg tif ... ok
testing sample.jpg tif[compression=jpeg] ... ok
testing sample.jpg tif[compression=deflate] ... ok
testing sample.jpg tif[compression=packbits] ... ok
testing sample.jpg tif[compression=jpeg,tile] ... ok
testing sample.jpg tif[compression=jpeg,tile,pyramid] ... ok
support for pngload not configured, skipping test
testing sample.jpg jpg ... ok
support for webpload not configured, skipping test
testing sample.jpg ppm ... ok
testing sample.jpg pfm ... ok
support for fitsload not configured, skipping test
testing mono.v csv ... ok
testing cmyk.v jpg ... ok
testing cmyk.v tif ... ok
testing cmyk.v tif[compression=jpeg] ... ok
testing cmyk.v tif[compression=jpeg,tile] ... ok
testing cmyk.v tif[compression=jpeg,tile,pyramid] ... ok
testing rad.v hdr ... ok
testing mono.v raw ... ok
testing sample.jpg raw ... ok
support for pdfload not configured, skipping test
testing trans-x.gif gifload ... 
stderr:
VipsForeignLoad: "/home/ubuntu/vips-8.13.0/test/test-suite/images/trans-x.png" is not a known file format
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

4/8 stall              OK               2.05s
5/8 connections        FAIL             0.03s   exit status 1
>>> MALLOC_PERTURB_=179 /home/ubuntu/vips-8.13.0/build-dir/test/test_connections.sh
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
stderr:
VipsForeignSave: ".png" is not a known target format
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

6/8 descriptors        OK               0.11s
7/8 fuzz               OK               1.53s
8/8 threading          OK               3.81s

Summary of Failures:

2/8 seq         FAIL             0.03s   exit status 1
3/8 formats     FAIL             2.16s   exit status 1
5/8 connections FAIL             0.03s   exit status 1

Ok:                 5   
Expected Fail:      0   
Fail:               3   
Unexpected Pass:    0   
Skipped:            0   
Timeout:            0   

Full log written to /home/ubuntu/vips-8.13.0/build-dir/meson-logs/testlog.txt
FAILED: meson-test 
/usr/bin/meson test --no-rebuild --print-errorlogs
ninja: build stopped: subcommand failed.

Maybe i need to install some packages or add specific arguments to meson setup? I used these arguments --buildtype=release --libdir=lib

[gabrieleolmi]

The cannot register existing type error can probably be fixed if libvips is linked with -Wl,-z,nodelete, see: #142 (comment)

Hopefully that would also fix the memory leak you were seeing.

I built libvips 8.13 from source and i did some quick tests. Vips\Config::cacheSetMax() seems to work and no more GLib errors appear 😀.

[jcupitt]

Woo! Great, thank you for doing the testing.

[gabrieleolmi]

Errors when calling Vips\Config::shutDown();

[23-Jul-2022 23:02:06] WARNING: [pool www] child 89482 said into stderr: "(process:89482): GLib-CRITICAL **: 23:02:06.461: g_hash_table_lookup: assertion 'hash_table != NULL' failed"
[23-Jul-2022 23:02:06] WARNING: [pool www] child 89482 said into stderr: "(process:89482): GLib-CRITICAL **: 23:02:06.461: g_hash_table_lookup: assertion 'hash_table != NULL' failed"
[23-Jul-2022 23:02:06] WARNING: [pool www] child 89482 said into stderr: "(process:89482): GLib-CRITICAL **: 23:02:06.462: g_hash_table_lookup: assertion 'hash_table != NULL' failed"
[23-Jul-2022 23:02:06] WARNING: [pool www] child 89482 said into stderr: "(process:89482): GLib-CRITICAL **: 23:02:06.462: g_hash_table_lookup: assertion 'hash_table != NULL' failed"
[23-Jul-2022 23:02:06] WARNING: [pool www] child 89482 said into stderr: "(process:89482): GLib-CRITICAL **: 23:02:06.462: g_hash_table_lookup: assertion 'hash_table != NULL' failed"
[23-Jul-2022 23:02:06] WARNING: [pool www] child 89482 said into stderr: "(process:89482): GLib-CRITICAL **: 23:02:06.462: g_hash_table_lookup: assertion 'hash_table != NULL' failed"
[23-Jul-2022 23:02:06] WARNING: [pool www] child 89482 said into stderr: "(process:89482): GLib-CRITICAL **: 23:02:06.462: g_hash_table_lookup: assertion 'hash_table != NULL' failed"
[23-Jul-2022 23:02:06] WARNING: [pool www] child 89482 said into stderr: "(process:89482): GLib-CRITICAL **: 23:02:06.463: g_hash_table_lookup: assertion 'hash_table != NULL' failed"
[23-Jul-2022 23:02:06] WARNING: [pool www] child 89482 said into stderr: "(process:89482): GLib-CRITICAL **: 23:02:06.463: g_hash_table_lookup: assertion 'hash_table != NULL' failed"
[23-Jul-2022 23:02:06] WARNING: [pool www] child 89482 said into stderr: "(process:89482): GLib-CRITICAL **: 23:02:06.463: g_hash_table_insert_internal: assertion 'hash_table != NULL' failed"
[23-Jul-2022 23:02:06] WARNING: [pool www] child 89482 said into stderr: "(process:89482): GLib-CRITICAL **: 23:02:06.463: g_hash_table_lookup: assertion 'hash_table != NULL' failed"
[23-Jul-2022 23:02:06] WARNING: [pool www] child 89482 exited on signal 11 (SIGSEGV - core dumped) after 166.434485 seconds from start
[23-Jul-2022 23:02:06] NOTICE: [pool www] child 90288 started

[jcupitt]

You can't use libvips again after shutdown, and you can't init again. Only call it just before process exit.

[gabrieleolmi]

I called Vips\Config::shutDown(); at the end of the script. So i think it's the fastcgi process that keeps running.

[jcupitt]

Yes, fastcgi is a persistent process, Don't call shutdown if you use it.