Too much reclaimable data in DB

[ImTei]

Hi, I'm Tei from Sunnyside Labs(Test in Prod) who is building op-erigon. One of our users recently reported a non-uniform DB size growth when they were using op-erigon. The size of the DB is multiple of a normal DB of the same chain. We have investigated this case with them and found the following things:

• Abnormal DB size growth is only seen on nodes serving debug_traceCall RPCs.
• The DB has too much reclaimable(GC) data.

  mdbx_stat v0.12.0-71-g1cac6536 (2022-07-28T09:57:31+07:00, T-9a6d7e5b917e5fbd14dc51835fa749d092aa1d72)
  Running for /home/node/data/chaindata/...
  Environment Info
  Pagesize: 8192
  Dynamic datafile: 24576..8796093022208 bytes (+16777216/-33554432), 3..1073741824 pages (+2048/-4096)
  Current mapsize: 8796093022208 bytes, 1073741824 pages 
  Current datafile: 2738108760064 bytes, 334241792 pages
  Last transaction ID: 56984418
  Latter reader transaction ID: 56984418 (0)
  Max readers: 32116
  Number of reader slots uses: 7
  Garbage Collection
  Pagesize: 8192
  Tree depth: 3
  Branch pages: 18
  Leaf pages: 6869
  Overflow pages: 98879
  Entries: 176269
  Page Usage
  Total: 1073741824 100%
  Backed: 334241792 31.1%
  Allocated: 334240650 31.1%
  Remained: 739501174 68.9%
  Used: 129165138 12.0%
  GC: 205075512 19.1%
  Retained: 10 0.0%
  Reclaimable: 205075502 19.1%
  Available: 944576676 88.0%

The Used pages are equal to the normal DB. So the abnormal DB growth is because of its reclaimable data, but it seems GC is not working as expected. I read some similar issues and MDBX docs so I understand that this is caused by some long-running DB txs -- maybe due to debug trace RPCs, but not sure the root cause.

As some guides from similar issues, it seems we can reclaim those data by copying MDBX, but it couldn't be the solution for the long-running nodes on production.

So I wonder if there's any solution or ongoing improvement about this issue. Please let me know if I'm missing something. Thank you!

[AskAlexSharov]

You need somehow limit - how long debug_traceCall allowed to work. Thats it. Just don’t open read transaction for unlimited amount of time.

[ImTei]

@AskAlexSharov Thank you. I have 2 questions.

• Does it mean timeout on the application or infra layer? or is there some way to set a limit on erigon?
• Is there any other way to reclaim other than mdbx copy? Will it be reclaimed automatically?

[awskii]

Hey @ImTei

• somewhere in op-erigon code. Erigon itself doesnt have flag to limit this.
• Automatic reclaim is not supported by mdbx, you have to copy old data to new one.