Free image after processing

[jeyemwey]

Hi,

thank you for this awesome library. I aspire to use it in my application for "cropping" profile images to squares when they are not uploaded as squares.

But: The library does not seem to free its used memory when it's done processing which means my server hits memory limit after two or three users uploaded their image. (200MB to 1.5GB, depending on the image)

The code I used to resize images

Please remind me if I have missed something and there is an option to cleanup.

[esimov]

Hi, thanks for the compliment. My question is when did you ran out of memory? After the process completion or during processing? Also did you ran the the process in parallel with big images? I'm asking this because Go is garbage collected, which means it's not needed to free up the memory programatically, so once the process is finished the memory should be freed up by Go runtime. After the input i should investigate the issue more deeply.

[jeyemwey]

I did not run several processes in parallel, only in sequence. The input image i used was a 1000x600 .jpg image. I run out of memory after the nth process since the memory is not released.

We've enabled logging memory logging:

2019/06/28 09:51:06 Starting resize for image ./static/upload/334a083755fd99b7a1481b961800233a.jpg
Alloc = 144 MiB TotalAlloc = 175 MiB    Sys = 160 MiB   NumGC = 7
Alloc = 23 MiB  TotalAlloc = 332 MiB    Sys = 165 MiB   NumGC = 12
Alloc = 106 MiB TotalAlloc = 571 MiB    Sys = 165 MiB   NumGC = 17
Alloc = 120 MiB TotalAlloc = 728 MiB    Sys = 169 MiB   NumGC = 20
Alloc = 257 MiB TotalAlloc = 1009 MiB   Sys = 273 MiB   NumGC = 24
Alloc = 38 MiB  TotalAlloc = 1201 MiB   Sys = 273 MiB   NumGC = 28
Alloc = 146 MiB TotalAlloc = 1445 MiB   Sys = 273 MiB   NumGC = 32
Alloc = 175 MiB TotalAlloc = 1639 MiB   Sys = 273 MiB   NumGC = 35
Alloc = 167 MiB TotalAlloc = 1916 MiB   Sys = 273 MiB   NumGC = 39
Alloc = 168 MiB TotalAlloc = 2201 MiB   Sys = 356 MiB   NumGC = 41
Alloc = 192 MiB TotalAlloc = 2508 MiB   Sys = 356 MiB   NumGC = 43
Alloc = 205 MiB TotalAlloc = 2803 MiB   Sys = 356 MiB   NumGC = 45
Alloc = 215 MiB TotalAlloc = 3095 MiB   Sys = 356 MiB   NumGC = 47
Alloc = 218 MiB TotalAlloc = 3380 MiB   Sys = 356 MiB   NumGC = 49
Alloc = 340 MiB TotalAlloc = 3643 MiB   Sys = 373 MiB   NumGC = 50
Alloc = 330 MiB TotalAlloc = 3914 MiB   Sys = 385 MiB   NumGC = 52
[...]
Alloc = 1008 MiB        TotalAlloc = 30331 MiB  Sys = 1549 MiB  NumGC = 130
Alloc = 1240 MiB        TotalAlloc = 30563 MiB  Sys = 1549 MiB  NumGC = 130
Alloc = 1458 MiB        TotalAlloc = 30781 MiB  Sys = 1615 MiB  NumGC = 130
Alloc = 909 MiB TotalAlloc = 31001 MiB  Sys = 1616 MiB  NumGC = 131
Alloc = 1066 MiB        TotalAlloc = 31159 MiB  Sys = 1616 MiB  NumGC = 131
Alloc = 1299 MiB        TotalAlloc = 31392 MiB  Sys = 1616 MiB  NumGC = 131
Alloc = 875 MiB TotalAlloc = 31616 MiB  Sys = 1616 MiB  NumGC = 132
Alloc = 1096 MiB        TotalAlloc = 31837 MiB  Sys = 1616 MiB  NumGC = 132
Alloc = 1309 MiB        TotalAlloc = 32050 MiB  Sys = 1616 MiB  NumGC = 132
Alloc = 839 MiB TotalAlloc = 32265 MiB  Sys = 1616 MiB  NumGC = 133
Alloc = 989 MiB TotalAlloc = 32415 MiB  Sys = 1616 MiB  NumGC = 133
Alloc = 1178 MiB        TotalAlloc = 32604 MiB  Sys = 1616 MiB  NumGC = 133
Alloc = 1403 MiB        TotalAlloc = 32829 MiB  Sys = 1616 MiB  NumGC = 133
Alloc = 942 MiB TotalAlloc = 33047 MiB  Sys = 1616 MiB  NumGC = 134
Alloc = 1174 MiB        TotalAlloc = 33279 MiB  Sys = 1616 MiB  NumGC = 134
Alloc = 1398 MiB        TotalAlloc = 33503 MiB  Sys = 1616 MiB  NumGC = 134
Alloc = 964 MiB TotalAlloc = 33736 MiB  Sys = 1682 MiB  NumGC = 135
Alloc = 1187 MiB        TotalAlloc = 33959 MiB  Sys = 1682 MiB  NumGC = 135
Alloc = 1409 MiB        TotalAlloc = 34181 MiB  Sys = 1682 MiB  NumGC = 135
Alloc = 1621 MiB        TotalAlloc = 34392 MiB  Sys = 1748 MiB  NumGC = 135
Alloc = 998 MiB TotalAlloc = 34613 MiB  Sys = 1748 MiB  NumGC = 136
2019/06/28 09:53:47 Done.
Alloc = 1133 MiB        TotalAlloc = 34749 MiB  Sys = 1748 MiB  NumGC = 136
Alloc = 1133 MiB        TotalAlloc = 34749 MiB  Sys = 1748 MiB  NumGC = 136
Alloc = 1133 MiB        TotalAlloc = 34749 MiB  Sys = 1748 MiB  NumGC = 136
Alloc = 1133 MiB        TotalAlloc = 34749 MiB  Sys = 1748 MiB  NumGC = 136
Alloc = 1133 MiB        TotalAlloc = 34749 MiB  Sys = 1748 MiB  NumGC = 136
Alloc = 1133 MiB        TotalAlloc = 34749 MiB  Sys = 1748 MiB  NumGC = 136
Alloc = 1133 MiB        TotalAlloc = 34749 MiB  Sys = 1748 MiB  NumGC = 136
Alloc = 1133 MiB        TotalAlloc = 34749 MiB  Sys = 1748 MiB  NumGC = 136

There, you can see that the memory used for image processing is not freed.

[esimov]

Thanks, I will take a look.

[esimov]

I have checked, the issue you have reported. There is no clean way to deallocate the data from the heap, since this is done automatically by the garbage collector. The only thing I can do is to assign a nil value to each variable which will be reallocated during the process, but since the Resize function (where all the heavy operation is happening) is called in iterative manner, there is no need to reassign the nil value to each of the reused variables. The GC is running periodically and will clean up the memory heap.

What you can try instead is to call runtime.GC() after each iteration, since you are using your own fabricated script to process the images.

Here is a simple script i have created to process multiple images and show the allocated memory. https://gist.github.com/esimov/e3b011a08c7d3bfd29f897d43104aed1

This is the output produced:

Alloc = 0 MiB   TotalAlloc = 0 MiB  Sys = 66 MiB    NumGC = 0
Alloc = 391 MiB TotalAlloc = 10700 MiB  Sys = 600 MiB   NumGC = 125
Alloc = 0 MiB   TotalAlloc = 10700 MiB  Sys = 600 MiB   NumGC = 126
=============================
Alloc = 410 MiB TotalAlloc = 22184 MiB  Sys = 601 MiB   NumGC = 265
Alloc = 0 MiB   TotalAlloc = 22184 MiB  Sys = 601 MiB   NumGC = 266
=============================
Alloc = 1140 MiB    TotalAlloc = 55607 MiB  Sys = 1665 MiB  NumGC = 419
Alloc = 0 MiB   TotalAlloc = 55607 MiB  Sys = 1665 MiB  NumGC = 420
=============================
Alloc = 827 MiB TotalAlloc = 84041 MiB  Sys = 1665 MiB  NumGC = 574
Alloc = 0 MiB   TotalAlloc = 84041 MiB  Sys = 1665 MiB  NumGC = 575

You can observe that calling runtime.GC() after each iteration this will free up the allocated memory, but this does not means that TotalAlloc will drop.

[esimov]

Closing this issue.