search

grafana / pyroscope

Continuous Profiling Platform. Debug performance issues down to a single line of code
https://grafana.com/oss/pyroscope/
GNU Affero General Public License v3.0
9.87k stars 585 forks source link

/tmp/multipart-xxxxxxxxxx files don't expire #1665

Closed netamego closed 1 year ago

netamego commented 1 year ago

Hi,

Pyroscope generates /tmp/multipart-xxxxxxxxxx files that are not expired and can cause the /tmp filesystem to fill up.

What is the purpose of these files?. Seems that are generated when browse in pyroscope gui. Could be an option to rotate this files? for example no more than 50 files.

Thanks.

kolesnikovae commented 1 year ago

Hi @netamego!

If you use Go/Java push agents, I'm afraid you might have encountered multipart size limit: https://github.com/pyroscope-io/pyroscope/blob/04ab88a1d6c7c8fabefe5a46bf69c3137de25d06/pkg/convert/pprof/profile.go#L189-L196

Just in case: are these files owned by pyroscope user? I'm asking because /tmp/multipart-xxxxxxxxxx is the pattern that is used by the standard Go library (mime/multipart package), see https://github.com/golang/go/issues/20253.

I think we have two options:

  1. Limit the request body size, to ensure these files are never created.
  2. Cleaning up files with RemoveAll.

I think we need to do the latter (despite my TODO note).

/cc @petethepig

netamego commented 1 year ago

Hi,

Yes.These multipart files are owned by pyroscope user. Sorry but I don't understand the problem. We use pyroscope exclusively with pyroscope java agent.

These multipart files constantly fill the /tmp filesystem when we browse the flamegraphs in the gui.

Thanks.

eh-am commented 1 year ago

Hey! Just to be clear, it doesn't seem to be related to the pyroscope gui, but when ingesting big profiles.

Thanks @kolesnikovae for having a look at it!

kolesnikovae commented 1 year ago

Hi @netamego! The fix is on the way: https://github.com/pyroscope-io/pyroscope/pull/1678

In the meantime, before we release it you can use the PR build image: pyroscope/pyroscope-dev:git-8739907009cbd8b303b8a6569113a4c9127c164f