[dynamodb] Performance optimizations

ssalonen commented 5 years ago

This PR introduces several performance optimizations to DynamoDB Persistence Service:

buffering data in the PersistenceService, and writing it in batches on background. We get performance gains by writing data in batches but also reduce items QueueingThreadPoolExecutor.taskQueue pressure in GenericItem.notifyListeners.
previously several checks were done on each store, causing significant overhead on store and query operations: 1) connection check (listing tables), 2) checking table existence. These are now removed, table is created on error (ResourceNotFoundException).

In addition, I have implemented exponential back-off retry for writes.

I've been verified the changes in my local setup. Before I could see that QueueingThreadPoolExecutor.taskQueue size growed (temporarily, but still) up to 500-1500 when many items were persisted at the same time. Dealing with the queue took some noticeable time. With the improvements, the size remains close to zero all the time.

Signed-off-by: Sami Salonen ssalonen@gmail.com

ssalonen commented 5 years ago

With the old version, I could see that sometimes QueueingThreadPoolExecutor.taskQueue growed to enormous sizes (even 1M), leading to OoM.

It's hard to diagnose what is the root cause for these occasional issues but I think writing in batches should be able to deal with even larger backlogs of data.

ssalonen commented 5 years ago

Anything I can do to progress this?

9037568 commented 5 years ago

Sorry, @ssalonen . My free time has been very low lately. I'll try to look at this this weekend. I fear we're in danger of crossing over with Kai's work on removing the persistence layer in #5844.

ssalonen commented 5 years ago

I understand completely, you have enormous work load in this repo alone.

Oh I was not aware of #5844, I think it #5827 can be dropped for now since the bnd build system will make it easier to handle SDK as maven dependency.

kaikreuzer commented 5 years ago

@ssalonen Would be cool if you could help on getting dynamodb (incl. your improvements) ready in https://github.com/openhab/openhab2-addons/pull/5275 - this is one of the addons, which I didn't yet spend the work to make it compile with bnd as it has a huge list of included libs.

Feel free to create a PR against my branch of that PR - merging your improvements should be very easy as I didn't do any real changes to the source code.

ssalonen commented 5 years ago

@9037568 thank you for the review. Addressed the comments in 6900a08 & 0f3cf54.

9037568 commented 5 years ago

Thanks, @ssalonen !

kaikreuzer commented 5 years ago

@ssalonen Could you please have a look at https://ci.openhab.org/job/openHAB1-Addons/lastCompletedBuild/org.openhab.persistence$org.openhab.persistence.dynamodb.test/testReport/? It seems that this PR made one test fail now.

openhab / openhab1-addons

[dynamodb] Performance optimizations #5826