High Memory Usage with Persistent IPFS Node in Browser via Helia

[orwithout]

I have initialized an IPFS node in my browser using the Helia package with persistence enabled. As the peerstore grows (currently over 5,000 peers), the memory consumption has reached 1GB. Despite spending dozens of hours reviewing the Helia and js-libp2p documentation, I haven't found an effective way to manage these resources. Specifically, I am struggling with reducing the number of storage peers and managing the current memory usage.

Environment:

• Browser version: Google Chrome 123.0.6312, Microsoft Edge 123.2420
• OS: Windows 2022

Attempts to Resolve:

• Documentation Reviewed:

  • Helia Initialization Interface
  • Libp2p Initialization Options
  • Libp2p Limits

• Configurations Tried:

  
  import { IDBBlockstore } from 'blockstore-idb';
  import { IDBDatastore } from 'datastore-idb';
  import { createHelia } from 'helia';
  import { yamux } from '@chainsafe/libp2p-yamux'
  import { mplex } from '@libp2p/mplex'
  import { noise } from '@chainsafe/libp2p-noise'

const instantiateHeliaNode = async () => { const datastore = new IDBDatastore('/datastore2'); const blockstore = new IDBBlockstore('/blockstore2'); await datastore.open(); await blockstore.open(); const heliaInstance = await createHelia({ libp2p: { streamMuxers: [ yamux(), mplex() ], connectionEncryption: [ noise() ], connectionManager: { maxConnections: 200, minConnections: 4, maxIncomingPendingConnections: 100, inboundConnectionThreshold: 100 } }, peerStore: { persistence: false, threshold: 5 }, keychain: { pass: 'very-strong-password', dek: { hash: 'sha2-512', salt: 'at-least-16-char-long-random-salt', iterationCount: 2000, keyLength: 64 } }, datastore: datastore, blockstore: blockstore });

return heliaInstance; };

window.helia = await instantiateHeliaNode();

I am seeking advice on how to automatically clean up the number of stored peers or set a cap on it.

[orwithout]

I have continued to monitor the memory usage impact of the peerStore, and through some debugging, I've noticed a significant decrease in memory usage when I remove the following monitoring code:

setInterval(async () => {
    const peers = await helia.libp2p.peerStore.all();
    console.log(`Total number of store peers: ${peers.length}`);
    console.log('Connected peers count:', helia.libp2p.getPeers().length);
}, 5000);

I guess this may be due to the fact that the relevant functions will create new objects and the timing of GC... Continuous monitoring may not be advisable?

Despite this improvement, I remain concerned about the peerStore's potential continuous growth without explicit limits or clear options for management.

[SgtPooki]

@orwithout If you have a very large peer store, your setInterval is loading the entire content of the peerstore, that you have configured to store in new IDBDatastore('/datastore2'), every 5 seconds. So it makes sense that memory would show as consistently "high" in this case.

In order to not load all peers into memory, there are a few options:

1. datastore query: https://ipfs.github.io/js-stores/interfaces/interface_datastore.index.Query.html
2. onProgress events