Test case 1st refresh creating ~1.2M records, 2nd refresh updating ~1.2M records with no changes, with 3 persisters and 5 partitions (with this setup, 1 persister is slacking, since it get only 1 partitions assigned)
with use_ar_object => true
refresh
max saving 1 part
avg saving 1 part
total time
max memory
1st
1.53s
0.47s
28.5m
100MB
2nd
1.09s
0.38s
25.3m
105MB
with use_ar_object => false
refresh
max saving 1 part
avg saving 1 part
total time
max memory
1st
1.53s
0.47s
28.5m
100MB
2nd
1.06s
0.28s
25.15m
97MB
Evaluation
use_ar_object => false is a bit quicker, but it always updates everything, so it stresses the DB much more. Also by updating everything, we don't get reliable report about what was changed.
miq-bot
commented
5 years ago
Test case 1st refresh creating ~1.2M records, 2nd refresh updating ~1.2M records with no changes, with 3 persisters and 5 partitions (with this setup, 1 persister is slacking, since it get only 1 partitions assigned)
with use_ar_object => true
with use_ar_object => false
Evaluation
use_ar_object => false is a bit quicker, but it always updates everything, so it stresses the DB much more. Also by updating everything, we don't get reliable report about what was changed.