Closed sjdonado closed 1 year ago
Context These changes came from the issue:
The value of "offset" is out of range. It must be >= 0 && <= 17825792. Received 17825795
Which is returned by mongo when a search or update query exceeds 17.825.792 bytes.
Following that, another issue was found:
-31800: transaction is too large and will not fit in the storage engine cache
That happens when transaction is too large and will not fit in the storage engine cache.
Approach
To resolve them, it was created a separate collection pois to store each POI as a document and they are inserted in chunks.
pois
POI
Those insertedIds are stored in poiListSnapshots, keeping them indexable as planned in the initial architectural design.
insertedIds
poiListSnapshots
lodash was added for the operations:
lodash
_.chunk(poiList, POI_LIST_CHUNK_SIZE)
_.values(insertedIds)
_.flatten(chunckedPoiListInsertedIds)
Context These changes came from the issue:
Which is returned by mongo when a search or update query exceeds 17.825.792 bytes.
Following that, another issue was found:
That happens when transaction is too large and will not fit in the storage engine cache.
Approach
To resolve them, it was created a separate collection
poisto store eachPOIas a document and they are inserted in chunks.Those
insertedIdsare stored inpoiListSnapshots, keeping them indexable as planned in the initial architectural design.lodashwas added for the operations:_.chunk(poiList, POI_LIST_CHUNK_SIZE)_.values(insertedIds)_.flatten(chunckedPoiListInsertedIds)