search

gramineproject / gramine

A library OS for Linux multi-process applications, with Intel SGX support
GNU Lesser General Public License v3.0
586 stars 193 forks source link

[LibOS] experimental flock significantly increases file size #1811

Open fnerdman opened 6 months ago

fnerdman commented 6 months ago

Description of the problem

Running go-ethereum in Gramine with experimental flock support we experience an increase of file size (300-1000%) compared to not setting the flock for the database folders.

Steps to reproduce

No response

Expected results

No response

Actual results

No response

Gramine commit hash

v1.6

dimakuv commented 6 months ago

Looking at the PR that introduced flock() support, I don't see anything that could lead to increasing the file size: https://github.com/gramineproject/gramine/pull/1416

Can you debug a bit more:

  1. What are the additional contents in this file? Is it just holes filled with all-zeros?
  2. The increase in size is in comparison to no-locks-at-all situation?
  3. Did you add flocks yourself, or are you using an app that can enable/disable this (otherwise how would it work before)?
fnerdman commented 6 months ago

After doing some more digging, this seems to be related to a regression in gramine's tmpfs implementation rather than flock. The increased file size is not observable when running gramine 1.5, whereas gramine +1.6 uses up significiant more space compared to the folder on disk. Copying a folder containing a huge amount of files <=2MB which amounts to ~280GB of size into tmpfs, even with enclave_size=1024GB results in OOM. See attachment chaindata.txt for folder structure.

mkow commented 6 months ago

The only commit I can see since 1.5 which I think could influence this could be 1ea3e60c2b52430cce04c741f2b79c4dc1316ff8, assuming it introduces some memory leak. I didn't see anything other related to tmpfs, but maybe I just missed something... Could you check if this is the commit causing the regression? If not, could you do git bisect to find the culprit? (assuming one run doesn't take too much time)

dimakuv commented 5 months ago

@lead4good Any progress on debugging this? I don't see anything special in the https://github.com/gramineproject/gramine/commit/1ea3e60c2b52430cce04c741f2b79c4dc1316ff8 commit.

I also don't understand how tmpfs (which uses buffers in enclave memory) relates to the file sizes in database folders. Or do I misunderstand your application and which files exactly are increased in size?