High memory utilization when creating a database from a dump

[penberg]

An user on Discord reports 10 GB memory consumption for 500 MB database file when creating a database from a dump:

sqlite3 ./dev.db '.output ./dump.sql' '.dump'
turso db shell $REMOTE_DB_NAME < ./dump.sql

[sivukhin]

Before streaming parsing attempt to load 715MB dump to turso ended up with OOM (eating 12GB of RAM and 12GB of swap)

After streaming parsing same attempt finished with timeout and consumes ~6-7 GB at peak

Still a lot of things to do - but we're getting better!

[haaawk]

I need to fix turso db shell < db.sql for shared schemas so at the same time I will do some improvements around the memory usage

[haaawk]

So this is tricky because golang will use all the available memory to reduce the number of GC runs. It does not mean that much memory is actually needed. One can use GOMEMLIMIT to tell Golang to use less memory and then it in fact uses less memory at the expense of execution time that increases due to GC running more often.