The node keeps track of the local data objects in storage in a Map. This is acceptable for now but is not scalable, as it will grow memory usage linearly with the number of objects stored.
addDataObjectIdToCache() should check for existence of entry before adding
loadDataObjectIdCache() should be protected to only run once
pin/unpinDataObjectIdToCache() should check existence of entry before pinning/unpinning
getDataObjectIdFromCache() - lock before check for id (I previously suggested it was not necessary in PR) I was wrong.
getDataObjectIDs() returns an array. At some point this is going to be expensive and we will hit the JavaScript limit on size of array elements. (32-bit index so ~4Billion elements)
kdembler
commented
8 months ago
Background
The node keeps track of the local data objects in storage in a
Map. This is acceptable for now but is not scalable, as it will grow memory usage linearly with the number of objects stored.https://github.com/Joystream/joystream/blob/5d911ce1bd546da16c983c852203a2a6eac8d8ea/storage-node/src/services/caching/localDataObjects.ts#L21
Some room for improvement.
addDataObjectIdToCache()should check for existence of entry before addingloadDataObjectIdCache()should be protected to only run oncepin/unpinDataObjectIdToCache()should check existence of entry before pinning/unpinninggetDataObjectIdFromCache()- lock before check for id (I previously suggested it was not necessary in PR) I was wrong.getDataObjectIDs()returns an array. At some point this is going to be expensive and we will hit the JavaScript limit on size of array elements. (32-bit index so ~4Billion elements)