search

OpenSmalltalk / opensmalltalk-vm

Cross-platform virtual machine for Squeak, Pharo, Cuis, and Newspeak.
http://opensmalltalk.org/
Other
547 stars 110 forks source link

Memory leak #642

Closed smalltalking closed 2 years ago

smalltalking commented 2 years ago

I noticed that recent VMs leak a few hundred kilobytes of memory every 3-6 seconds. It happens even if you just start VM and do nothing with the image. I tested it with various 64-bit Linux VMs. There seems to be no upper limit on the leaked memory. The leak happens even with the -memory option set. The size of the object memory is not affected by the leak, saved images remain small and the image does not see the increase in memory usage. Affected VMs include: 5.0-202206021410, 5.0-202205110711, 5.0-202204190959 Unaffected VMs include: 5.0-202112022203 and 5.0-202112201228 The 5.0-202206021410 assert and debug VMs leak the memory as well.

OpenSmalltalk-Bot commented 2 years ago

Ubuntu 20.04.4 LTS. Start the image and watch as the resident set size grows indefinitely:

$ watch ps -C squeak -o rss

You may prefer htop over watch+ps.

Levente

On Sun, 19 Jun 2022, tim Rowledge wrote:

Which platform? How are you measuring it? I'm not seeing anything that makes me suspect this on pi/linux.

On 2022-06-19, at 2:11 PM, smalltalking @.***> wrote:

I noticed that recent VMs leak a few hundred kilobytes of memory every 3-6 seconds. It happens even if you just start VM and do nothing with the image. I tested it with various 64-bit Linux VMs. There seems to be no upper limit on the leaked memory. The leak happens even with the -memory option set. The size of the object memory is not affected by the leak, saved images remain small and the image does not see the increase in memory usage. Affected VMs include: 5.0-202206021410, 5.0-202205110711, 5.0-202204190959 Unaffected VMs include: 5.0-202112022203 and 5.0-202112201228 The 5.0-202206021410 assert and debug VMs leak the memory as well.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.

tim

tim Rowledge; @.***; http://www.rowledge.org/tim Useful Latin Phrases:- Noli me vocare, ego te vocabo = Don't call me, I'll call you.

OpenSmalltalk-Bot commented 2 years ago

Eeek. That's nasty.

On Raspberry Pi OS the number does go up, but at worst very slowly, and seems to go back down as well.

On an x64 Ubuntu 20.04.1 machine of my acquaintance :-

uname -a Linux thing 5.13.0-44-generic #49~20.04.1-Ubuntu SMP Wed May 18 18:44:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Ah, but an older VM, so providing a backstop version :-

Virtual Machine

/usr/bin/sageteacloudsq/lib/squeak Open Smalltalk Cog[Spur] VM [CoInterpreterPrimitives VMMaker.oscog-eem.2936] Unix built on Mar 31 2021 18:28:22 Compiler: 4.2.1 Compatible Clang 6.0.0 (tags/RELEASE_600/final) platform sources revision VM: 202101191919 @.:DizietFS/Documents/Squeak/opensmalltalk-vm Date: Tue Jan 19 11:19:12 2021 CommitHash: 253cbfc14 Plugins: 202101191919 @.:DizietFS/Documents/Squeak/opensmalltalk-vm CoInterpreter VMMaker.oscog-eem.2936 uuid: 69233536-ece4-4ea1-b523-c28c4aa7d4c2 Mar 31 2021 StackToRegisterMappingCogit VMMaker.oscog-eem.2936 uuid: 69233536-ece4-4ea1-b523-c28c4aa7d4c2 Mar 31 2021

tim

tim Rowledge; @.***; http://www.rowledge.org/tim Fractured Idiom:- MERCI RIEN - Thanks for nothin'.

eliotmiranda commented 2 years ago

On Sun, Jun 19, 2022 at 2:11 PM smalltalking @.***> wrote:

I noticed that recent VMs leak a few hundred kilobytes of memory every 3-6 seconds. It happens even if you just start VM and do nothing with the image.

Can someone bisect and identify the commit that introduced this?

I tested it with various 64-bit Linux VMs. There seems to be no upper limit on the leaked memory. The leak happens even with the -memory option set. The size of the object memory is not affected by the leak, saved images remain small and the image does not see the increase in memory usage. Affected VMs include: 5.0-202206021410, 5.0-202205110711, 5.0-202204190959 Unaffected VMs include: 5.0-202112022203 and 5.0-202112201228 The 5.0-202206021410 assert and debug VMs leak the memory as well.

,,,^..^,,, best, Eliot

marceltaeumel commented 2 years ago

I cannot reproduce this issue on

OS: Win32 10.0 X64
VM: 202206021410 (c9fd365)
SQ: Squeak6.0beta #22056 (64 bit)

Yet, I can reproduce it on Ubuntu 18.04 👍

edoneel commented 2 years ago

So I can trigger it so I'll try to bisect it.

Squeak6.0beta latest update: #22053

platform sources revision VM: 202206181934 XXX@YYY/opensmalltalk-vm Date: Sat Jun 18 12:34:41 2022 CommitHash: 27657fe6a Plugins: 202206181934 XXX@YYY:tmp/opensmalltalk-vm CoInterpreter VMMaker.oscog-eem.3188 uuid: 010bab73-d814-4c05-b0f1-f2da78c1568f Jun 20 2022 StackToRegisterMappingCogit VMMaker.oscog-eem.3188 uuid: 010bab73-d814-4c05-b0f1-f2da78c1568f Jun 20 2022

Ubuntu 22.04 - x86-64, does it, see below

Raspberry PI ARMv8 64 bit based on Debian 11.3, ie, April 4 2022, does not.

Raspberry PI ARMv7 32 bit based on Debian 11.3, ie, April 4 2022 does not. Of course this one is the 32 bit VM and the 32 bit image but the identical image/vm combo as above.

Ubuntu 18.04 ARMv8 64 bit does it, also see below.

========================== Ubuntu 22.04 - x86-64.

$ while true; do ps -C squeak -o rss; sleep 5; done RSS 71668 RSS 71928 RSS 72984 RSS 73248 RSS 73248 RSS 73512 RSS 73776 RSS 73776 RSS 74040 RSS 74040 RSS 74304 RSS 74304 RSS 74568 RSS 74568 RSS 74832 RSS 75096 RSS 75096 RSS 75360 RSS 75360 RSS 75624 RSS 75624 RSS 75888 RSS 76152 RSS 76152

===================================== Ubuntu 18.04, ARMv8 64 bit

76764 RSS 76764 RSS 77028 RSS 77292 RSS 77292 RSS 77556 RSS 77556 RSS 77820 RSS 77820 RSS 78084 RSS 78348 RSS 78348 RSS 78612 RSS 78612 RSS 78876 RSS 78876 RSS 79140 RSS 79404 RSS 79404 RSS 79668 RSS 79668 RSS 79932 RSS 79932 RSS 80196 RSS 80460

edoneel commented 2 years ago

So I spent some time looking into this and I think I have tracked down when it happens. In all cases my test case was Ubuntu 22.04 on x86-64 but I also see it on Ubuntu 18.04 on ARMv8.

Some recent change to the combo of the VM and the image will do this, but, only on a local X11 display when you are displaying. Using -vm-display-X11 -headless prevents this from happening, so, good news, servers will be ok. It also does not happen when you use a remote X11 display. It only happens when you use a local graphics card.

It does not show up in the squeak memory usage, just the RSS size. I guess that something is not getting freed somewhere in C, and it looks to be somewhere in the C code for the X11 driver.

Wayland vs X11 does not change anything.

This was not in Squeak 5.3 and the Squeak 53 VM running almost the newest 6.0 image is ok too.

e908c9c22 20220621T1716 Squeak6.0rc1 22076 Bad 202206021410 Squeak6.0rc1 22076 Bad e908c9c22 20220621T1716 Squeak5.3-19481 Good 6a0bc96 202003021730 Squeak6.0beta-21757 Good

edoneel commented 2 years ago

This can be closed here because it is not a VM issue. It is an image side issue between image update 20981, which is good, and image update 21617 which has this problem. Updating a working 20981 image with all updates causes it to start increasing RSS.

edoneel commented 2 years ago

Closed relative to https://github.com/squeak-smalltalk/squeak-object-memory/issues/51

edoneel commented 2 years ago

I think this might be better solved on the VM side.

edoneel commented 2 years ago

So - Sometime this spring between change 20981 and change 21617 there were changes made to the screen resolution code that changed how often, it seems, we call display_ioScreenScaleFactor. Before it must have been rarely called like on startup and resize or whatever and now it is called often, sometimes many times per second.

The X11 code, being X11 code for Linux, has multiple ways of getting the screen scale factor. Which path was followed depended on some things but on assorted Ubuntu systems the path followed called scale_Xftdpi_usable. This called XrmGetStringDatabase every time it entered but if Xft.dpi was there then it returned without calling XrmDestroyDatabase. Ergo the memory leaked.

The patch is trivial and attached

PatchX11Scale.patch.txt .

krono commented 2 years ago

Oh, how embarrasing. I'm really sorry.

marceltaeumel commented 2 years ago

This can be closed here because it is not a VM issue. It is an image side issue between image update 20981, which is good, and image update 21617 which has this problem. Updating a working 20981 image with all updates causes it to start increasing RSS.

Just a minor remark on this intermediate conclusion. At that point, we already knew that the .image file was not affected. It did not get bigger because of the memory leak. While an image-side workaround might have been possible, it clearly pointed to VM-side code not being correct. From the top of my hat, I cannot think of a scenario where non-FFI, image-only code leaks memory without affecting the .image size. (Maybe thousands of open file handles? But those have image-side counterparts as well, thus affecting the .image size.)

Thanks for the thorough investigation, Bruce!!! ❤️

marceltaeumel commented 2 years ago

Documented here:

Squeak-app bundles use workaround for now:

edoneel commented 1 year ago

Hi all,

So I can trigger it so I'll try to bisect it.

cheers

bruce

Squeak6.0beta

latest update: #22053

platform sources revision VM: 202206181934 @./opensmalltalk-vm Date: Sat Jun 18 12:34:41 2022 CommitHash: 27657fe6a Plugins: 202206181934 @.:tmp/opensmalltalk-vm

CoInterpreter VMMaker.oscog-eem.3188 uuid: 010bab73-d814-4c05-b0f1-f2da78c1568f Jun 20 2022

StackToRegisterMappingCogit VMMaker.oscog-eem.3188 uuid: 010bab73-d814-4c05-b0f1-f2da78c1568f Jun 20 2022

Ubuntu 22.04 - x86-64, does it, see below

Raspberry PI ARMv8 64 bit based on Debian 11.3, ie, April 4 2022, does not.

Raspberry PI ARMv7 32 bit based on Debian 11.3, ie, April 4 2022 does not.  Of course this one is the 32 bit VM and the 32 bit image but the identical image/vm combo as above.

Ubuntu 18.04 ARMv8 64 bit does it, also see below.

==========================

Ubuntu 22.04 - x86-64.

$ while true; do ps -C squeak -o rss; sleep 5; done

  RSS

71668

  RSS

71928

  RSS

72984

  RSS

73248

  RSS

73248

  RSS

73512

  RSS

73776

  RSS

73776

  RSS

74040

  RSS

74040

  RSS

74304

  RSS

74304

  RSS

74568

  RSS

74568

  RSS

74832

  RSS

75096

  RSS

75096

  RSS

75360

  RSS

75360

  RSS

75624

  RSS

75624

  RSS

75888

  RSS

76152

  RSS

76152

=====================================

Ubuntu 18.04, ARMv8 64 bit

76764

  RSS

76764

  RSS

77028

  RSS

77292

  RSS

77292

  RSS

77556

  RSS

77556

  RSS

77820

  RSS

77820

  RSS

78084

  RSS

78348

  RSS

78348

  RSS

78612

  RSS

78612

  RSS

78876

  RSS

78876

  RSS

79140

  RSS

79404

  RSS

79404

  RSS

79668

  RSS

79668

  RSS

79932

  RSS

79932

  RSS

80196

  RSS

80460

On 2022-06-22T08:18:17.000+02:00, Marcel Taeumel @.***> wrote:

 I cannot reproduce this issue on    OS: Win32 10.0 X64  VM: 202206021410 (c9fd365)  SQ: Squeak6.0beta #22056 (64 bit)    —  Reply to this email directly, view it on GitHub  [https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/642#issuecomment-1162690028],  or unsubscribe  [https://github.com/notifications/unsubscribe-auth/ACEXJISYDDVRRBJHGY6M6XTVQKV2TANCNFSM5ZG6PKXA].  You are receiving this because you are subscribed to this  thread.Message ID: @.***>

OpenSmalltalk-Bot commented 1 year ago

Which platform? How are you measuring it? I'm not seeing anything that makes me suspect this on pi/linux.

On 2022-06-19, at 2:11 PM, smalltalking @.***> wrote:

I noticed that recent VMs leak a few hundred kilobytes of memory every 3-6 seconds. It happens even if you just start VM and do nothing with the image. I tested it with various 64-bit Linux VMs. There seems to be no upper limit on the leaked memory. The leak happens even with the -memory option set. The size of the object memory is not affected by the leak, saved images remain small and the image does not see the increase in memory usage. Affected VMs include: 5.0-202206021410, 5.0-202205110711, 5.0-202204190959 Unaffected VMs include: 5.0-202112022203 and 5.0-202112201228 The 5.0-202206021410 assert and debug VMs leak the memory as well.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.

tim

tim Rowledge; @.***; http://www.rowledge.org/tim Useful Latin Phrases:- Noli me vocare, ego te vocabo = Don't call me, I'll call you.