jankotek
commented
7 years ago It is a design mistake in MapDB. In older version the catalog was BTreeMap which was not fully loaded. Will fix it in next version Jan On Wed, 2017-04-12 at 21:26 -0700, Kosenko Pavel Viktorovich wrote:
Hi ))
There are need to create about 4k maps for one of subsystem of our production code.
Every map creation lead to serialization and deserialization of full catalog of stored maps description. Db.nameCatalogLoad() and DB.nameCatalogSave() functions takes progressively more time as tables count grow (same as naive string concatenation lead to exponential time consuming to progress).
Creating separate file for each map is slow (because of OS filesystem metadata synchronization) and too space consuming for small tables.
Can be optimized operation of updating catalog (cached in memory for example and flushed to disk by method invocation or by timer callback) or updated by parts without need to serialize/deserialize and save/load whole catalog at once?
Big thanks for great library!
Our system use it (previous 2.0 version) for store millions of objects offheap. But now we have some problems with big cost of too many WeakReference objects (were in 2.0 version) that leads to long GC pauses. Now I'm trying to adopt new version of library and encountered several problems. Most critical I have written above.
Second problem is absent of method for delete created map from db, which is not critical as I can clear/recycle already created tables for reuse.
Sorry for English which is not my native language. ))
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.
{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c5 5493e4bb","name":"GitHub"},"entity":{"external_key":"github/jankotek/ mapdb","title":"jankotek/mapdb","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/ass ets/143418/17495839/a5054eac-5d88-11e6-95fc- 7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent .com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed- b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/jankotek/mapdb"}},"updates":{"snipp ets":[{"icon":"DESCRIPTION","message":"Too slow many maps in single database creation (#821)"}],"action":{"name":"View Issue","url":"https://github.com/jankotek/mapdb/issues/821"}}}
Hi ))
There are need to create about 4k maps for one of subsystem of our production code. Every map creation lead to serialization and deserialization of full catalog of stored maps description. Db.nameCatalogLoad() and DB.nameCatalogSave() functions takes progressively more time as tables count grow (same as naive string concatenation lead to exponential time consuming to progress). Creating separate file for each map is slow (because of OS filesystem metadata synchronization) and too space consuming for small tables.
Can be optimized operation of updating catalog (cached in memory for example and flushed to disk by method invocation or by timer callback) or updated by parts without need to serialize/deserialize and save/load whole catalog at once?
Big thanks for great library! Our system use it (previous 2.0 version) for store millions of objects offheap. But now we have some problems with big cost of too many WeakReference objects (were in 2.0 version) that leads to long GC pauses. Now I'm trying to adopt new version of library and encountered several problems. Most critical I have written above. Second problem is absent of method for delete created map from db, which is not critical as I can clear/recycle already created tables for reuse.
Sorry for English which is not my native language. ))