search

iTwin / imodel-transformer

API for exporting an iModel's parts and also importing them into another iModel
MIT License
3 stars 2 forks source link

Start using BigMap instead of regular Map #85

Closed simsta6 closed 1 year ago

simsta6 commented 1 year ago

When transforming iModels with huge amounts of Element Aspects process can crash.

BigMap implementation will decrease performance by little bit.

To decrease memory usage, we can use https://www.npmjs.com/package/lmdb

MichaelBelousov commented 1 year ago

BigMap implementation will decrease performance by little bit.

Have you done any testing to quantify the difference? @DanRod1999 can you run the performance tests on this branch?

MichaelBelousov commented 1 year ago

internal performance report says 2-3% increase in transformation time. Let's try the per-last-char-hashed bucket imo, but otherwise this is fine for a short term fix. Longer term fix will be trying to use a compressed map #83

KyryloVolotovskyi commented 1 year ago

internal performance report says 2-3% increase in transformation time. Let's try the per-last-char-hashed bucket imo, but otherwise this is fine for a short term fix. Longer term fix will be trying to use a compressed map #83

How many entities did the iModels that were used for testing have? If the amount was rather small then there shouldn't be a noticeable impact on the overall performance. Can 2-3% increase just be some noise?

MichaelBelousov commented 1 year ago

How many entities did the iModels that were used for testing have? If the amount was rather small then there shouldn't be a noticeable impact on the overall performance. Can 2-3% increase just be some noise?

@DanRod1999 can you answer how many entities are in the test iModels? Multiple iModels saw similar performance downgrades in the tests, and they are not small iModels. I do not think it was a fluke. Granted, my sample size is small, but when the tests run as long we have them, on those well-calibrated performance test machines, I think it's unlikely that was noise

DanRod1999 commented 1 year ago

How many entities did the iModels that were used for testing have? If the amount was rather small then there shouldn't be a noticeable impact on the overall performance. Can 2-3% increase just be some noise?

@DanRod1999 can you answer how many entities are in the test iModels? Multiple iModels saw similar performance downgrades in the tests, and they are not small iModels. I do not think it was a fluke. Granted, my sample size is small, but when the tests run as long we have them, on those well-calibrated performance test machines, I think it's unlikely that was noise

I don't know for sure but I can say the GB size for them are both less than 2GB. I believe the imodels I have been using are 1.2 and .87 gb. I will try to find the entity counts