helium / blockchain-core

Apache License 2.0
213 stars 85 forks source link

Fix ledger context leaks in `poc_receipts_v1` and `rewards_v2` #1372

Closed xandkar closed 2 years ago

xandkar commented 2 years ago

Part of the bigger memory leaks hunt during resync.

Methodology

  1. Spotted ETS table growths using beam_stats.
  2. Instrumented:
    • blockchain_ledger_v1:new_context/1 and
    • blockchain_ledger_v1:delete_context/1 to log the generated table references and stack traces.
  3. Processed the resulting logs to find origins (i.e. stacktraces) of references that did not have a corresponding delete_context call:

    $ grep -IHn 'Ledger context' miner/_build/resync_test/rel/miner/log/warning* \
    | awk '
    {
        op = $7;
        ref = $8;
        stack = $11;
    
        if (op == "DELETED")
            del[ref] = stack;
        else if (op == "CREATED")
            new[ref] = stack
    }
    
    END {
        for (ref in new)
            if (!del[ref])
                leaks[new[ref]]++;
        for (stack in leaks)
            printf "%d %s\n\n", leaks[stack], stack
    }'