search

Granola-Team / mina-indexer

The Mina Indexer is a re-imagined version of the software collectively called the "Mina archive node."
Apache License 2.0
18 stars 10 forks source link

Problem: restoring a large snapshot fails due to being OOM-killed #1283

Closed robinbb closed 2 months ago

robinbb commented 2 months ago

Creating a snapshot is an operation performed regularly in tier-3 CI, but restoring that snapshot is not part of an automated process, at present. When I attempted to restore a snapshot of size 58425524989, the process began consuming RAM until the VMM consumption was 34 GB (32 GB RAM on the machine + some swap space), and then was killed by the OOM-killer (as discoverable by using journalctl -xe to examine the logs). On the command line, this looks like:

> bin/mina-indexer-adc1657e database restore --snapshot-file snapshots/0.7.1-359604-adc1657e.snapshot --restore-dir ./tmp-restore                  
2024-07-10T15:50:34.377695+00:00 - INFO Restoring mina indexer database from snapshot file "snapshots/0.7.1-359604-adc1657e.snapshot" to "./tmp-restore"                                      
zsh: killed     bin/mina-indexer-adc1657e database restore --snapshot-file  --restore-dir
robinbb commented 2 months ago

Without a fix for this, we cannot make use of the snapshot functionality to deploy databases to production.