ipfs / helia

An implementation of IPFS in JavaScript
https://helia.io
Other
811 stars 81 forks source link

High Memory Usage with Persistent IPFS Node in Browser via Helia #526

Open orwithout opened 2 months ago

orwithout commented 2 months ago

I have initialized an IPFS node in my browser using the Helia package with persistence enabled. As the peerstore grows (currently over 5,000 peers), the memory consumption has reached 1GB. Despite spending dozens of hours reviewing the Helia and js-libp2p documentation, I haven't found an effective way to manage these resources. Specifically, I am struggling with reducing the number of storage peers and managing the current memory usage.

Environment:

Attempts to Resolve:

const instantiateHeliaNode = async () => { const datastore = new IDBDatastore('/datastore2'); const blockstore = new IDBBlockstore('/blockstore2'); await datastore.open(); await blockstore.open(); const heliaInstance = await createHelia({ libp2p: { streamMuxers: [ yamux(), mplex() ], connectionEncryption: [ noise() ], connectionManager: { maxConnections: 200, minConnections: 4, maxIncomingPendingConnections: 100, inboundConnectionThreshold: 100 } }, peerStore: { persistence: false, threshold: 5 }, keychain: { pass: 'very-strong-password', dek: { hash: 'sha2-512', salt: 'at-least-16-char-long-random-salt', iterationCount: 2000, keyLength: 64 } }, datastore: datastore, blockstore: blockstore });

return heliaInstance; };

window.helia = await instantiateHeliaNode();



I am seeking advice on how to automatically clean up the number of stored peers or set a cap on it.
orwithout commented 1 month ago

I have continued to monitor the memory usage impact of the peerStore, and through some debugging, I've noticed a significant decrease in memory usage when I remove the following monitoring code:

setInterval(async () => {
    const peers = await helia.libp2p.peerStore.all();
    console.log(`Total number of store peers: ${peers.length}`);
    console.log('Connected peers count:', helia.libp2p.getPeers().length);
}, 5000);

I guess this may be due to the fact that the relevant functions will create new objects and the timing of GC... Continuous monitoring may not be advisable?

Despite this improvement, I remain concerned about the peerStore's potential continuous growth without explicit limits or clear options for management.