search

cojen / TuplDB

TuplDB is a high-performance, concurrent, transactional, scalable, low-level embedded database.
GNU Affero General Public License v3.0
114 stars 23 forks source link

Time to open database? #64

Closed amcp closed 7 years ago

amcp commented 7 years ago

I am running some tests on version 1.3.10.3. I open databases with the Database.open(config) call. I notice that this call takes around 5 seconds with the configuration below. Is there anything I can do to speed up time to start up? I have a test suite that runs around ~300 test cases and am trying to see if I can speed them up.

mBaseFile = {File@1299} "/Users/amcp/git/tupl-titan-storage-backend/target/db/tupl" mMkdirs = true mDataFiles = null mMapDataFiles = false mDataPageArray = null mFileFactory = null mMinCachedBytes = 100000000 mMaxCachedBytes = 0 mSecondaryCacheSize = 0 mDurabilityMode = {DurabilityMode@1284} "SYNC" mLockUpgradeRule = {LockUpgradeRule@1285} "STRICT" mLockTimeoutNanos = 60000000000 mCheckpointRateNanos = 1000000000 mCheckpointSizeThreshold = 1073741824 mCheckpointDelayThresholdNanos = 60000000000 mEventListener = null mFileSync = false mReadOnly = false mPageSize = 4096 mDirectPageAccess = {Boolean@1300} "true" mCachePriming = false mReplManager = null mMaxReplicaThreads = 0 mCrypto = null mTxnHandler = null mReplRecoveryStartNanos = 0 mReplInitialTxnId = 0

broneill commented 7 years ago

If you're just running tests, then swap the min/max cache settings. I suspect the memory allocation is causing the delay. Also, use the NO_FLUSH durability mode to speed things up for testing.

amcp commented 7 years ago

Makes sense. Now, I am using 1MB for min cache and 1GB for max cache. And I changed the durability mode to NO_FLUSH. Still taking 5 seconds to open the DB. Below is the updated configuration. I will try this on EC2 momentarily...

mBaseFile = {File@1299} "/Users/amcp/git/tupl-titan-storage-backend/target/db/tupl" mMkdirs = true mDataFiles = null mMapDataFiles = false mDataPageArray = null mFileFactory = null mMinCachedBytes = 1048576 mMaxCachedBytes = 1073741824 mSecondaryCacheSize = 0 mDurabilityMode = {DurabilityMode@1284} "NO_FLUSH" mLockUpgradeRule = {LockUpgradeRule@1285} "STRICT" mLockTimeoutNanos = 60000000000 mCheckpointRateNanos = 1000000000 mCheckpointSizeThreshold = 1073741824 mCheckpointDelayThresholdNanos = 60000000000 mEventListener = null mFileSync = false mReadOnly = false mPageSize = 4096 mDirectPageAccess = {Boolean@1300} "true" mCachePriming = false mReplManager = null mMaxReplicaThreads = 0 mCrypto = null mTxnHandler = null mReplRecoveryStartNanos = 0 mReplInitialTxnId = 0

amcp commented 7 years ago

commands to elicit timings: https://github.com/amcp/tupl-titan-storage-backend/tree/janus#run-all-tests-against-tupl-on-an-ec2-amazon-linux-ami

checkout janus and build it: https://github.com/JanusGraph/janusgraph.git: mvn install -DskipTest=true checkout the janus branch in titan-tupl replace the last mvn test command with the following: mvn test -Pintegration-tests -Dtest=TuplFixedLengthKCVSTest#storeAndRetrieveWithClosing

amcp commented 7 years ago

EC2 gives much more reasonable results: Time to open store manager: 227 Time to clear storage: 30 Time to close store manager: 8 Time to open: 11 Time to generate values: 20 Time to load values: 92 Time to closeopen: 40 Time to check value existence: 87 Time to check value existence: 57

amcp commented 7 years ago

I guess there is an issue with my computer. Sorry for the trouble! Alex