Memory management using Lilliput

[thoas]

Hi,

We are using lilliput as custom backend in picfit, it's a better way to handle gif than golang image standard library.

Since Lilliput handles memory using buffer, the allocated memory of the image server is increasing over time with the following config:

// DefaultMaxBufferSize is the maximum size of buffer for lilliput
const DefaultMaxBufferSize = 8192

// DefaultImageBufferSize is the default image buffer size for lilliput
const DefaultImageBufferSize = 50 * 1024 * 1024

What approach do you recommend in this case?

Thank you

[brian-armstrong-discord]

Hi thoas,

I'm not sure what you mean by Since Lilliput handles memory using buffer, the allocated memory of the image server is increasing over time with the following config:. In our usage, the memory usage doesn't seem to increase over time. Can you give me more details about your situation?

[brian-armstrong-discord]

I looked at the code again and it looks like you might be allocating a new Ops each time you transform an image. You will most likely want to create a pool of these at the start of your program and then keep them around forever.

[thoas]

Thank you for your review @brian-armstrong-discord

Based on your experience, do you have a similar workflow? If so, what's the length of your pool?

A detailed view of a pprof in our production instance:

(pprof) top20
Showing nodes accounting for 512MB, 99.61% of 514MB total
Dropped 36 nodes (cum <= 2.57MB)
Showing top 20 nodes out of 23
      flat  flat%   sum%        cum   cum%
     512MB 99.61% 99.61%      512MB 99.61%  github.com/discordapp/lilliput.NewFramebuffer (inline)
         0     0% 99.61%      512MB 99.61%  github.com/discordapp/lilliput.NewImageOps (inline)
         0     0% 99.61%      512MB 99.61%  github.com/gin-gonic/contrib/sentry.Recovery.func1
         0     0% 99.61%      512MB 99.61%  github.com/gin-gonic/gin.(*Context).Next
         0     0% 99.61%      512MB 99.61%  github.com/gin-gonic/gin.(*Engine).ServeHTTP
         0     0% 99.61%      512MB 99.61%  github.com/gin-gonic/gin.(*Engine).handleHTTPRequest
         0     0% 99.61%      512MB 99.61%  github.com/gin-gonic/gin.LoggerWithWriter.func1
         0     0% 99.61%      512MB 99.61%  github.com/thoas/picfit/application.ImageFileFromContext
         0     0% 99.61%      512MB 99.61%  github.com/thoas/picfit/application.processImage
         0     0% 99.61%      512MB 99.61%  github.com/thoas/picfit/engine.Engine.Transform
         0     0% 99.61%      512MB 99.61%  github.com/thoas/picfit/engine.operate
         0     0% 99.61%      512MB 99.61%  github.com/thoas/picfit/engine/backend.(*Lilliput).Thumbnail
         0     0% 99.61%      512MB 99.61%  github.com/thoas/picfit/engine/backend.(*Lilliput).transform
         0     0% 99.61%      512MB 99.61%  github.com/thoas/picfit/middleware.KeyParser.func1
         0     0% 99.61%      512MB 99.61%  github.com/thoas/picfit/middleware.OperationParser.func1
         0     0% 99.61%      512MB 99.61%  github.com/thoas/picfit/middleware.ParametersParser.func1
         0     0% 99.61%      512MB 99.61%  github.com/thoas/picfit/middleware.RestrictSizes.func2
         0     0% 99.61%      512MB 99.61%  github.com/thoas/picfit/middleware.Security.func1
         0     0% 99.61%      512MB 99.61%  github.com/thoas/picfit/middleware.URLParser.func1
         0     0% 99.61%      512MB 99.61%  github.com/thoas/picfit/middleware/context.SetContext.func1

[brian-armstrong-discord]

@thoas You will probably want something like one Ops per CPU, or fewer if your boxes are memory constrained.

[thoas]

@brian-armstrong-discord I'm closing the issue since it's only related to our usage.

I have appreciated that you have spent your time answering my questions, thank you for your input.