Persistence - Performance Improvement

[sageserpent-open]

Improve performance of the persistent implementation introduced in #16.

[sageserpent-open]

In commit c7836ff82ef5b6fcfc8bb9d1628af79f6c6a94e1, the idea was to aggressively proxy Scala objects by scraping out their interfaces and generating proxies that subclass the interface, rather than the actual runtime type, thus working around the problems caused by practically everything in Scala having a final runtime type.

That way, the intention is to not have to hop so much between tranches when performing queries as hopefully many of the proxies will not have to load their underlying objects.

In commit 6b3d4159190c351dc344c510ab544d2e4ea8a16a, this has been extended by barring inter-tranche references for all objects that cannot be proxied - there are lots of instances of 'Class' that fall into that category, so these are now stored locally in the tranche, even if they are duplicated across many tranches (They are still reference counted within each tranche, though).

While this works, the results are still uninspiring:

[sageserpent-open]

It is odd that querying using the scope that refers to the latest revision should cause so much activity. After all, the latest revision should have its associated tranche in the cache, so everything should be to hand to get the blob storage.

Perhaps a better tack would be to look at either:

1. Breaking out blob storage from the timeline and storing it with its own tranche id, so tranches aren't so big (and interdependent). Note that while blob storage objects are supposed to share internal structure, they will be placed within separate timeline implementations - so the corresponding tranches will carry a lot of baggage. The same goes for the all events objects too. Hmmm...

2. Pithing out blob storage altogether by using some other mechanism to store its data. The blob storage object would still be stored as part of a tranche, but its internals would not. This has a lot of appeal, the clue being in the name 'BlobStorage' - why should it have to float on top of 'ImmutableObjectStorage'.

3. Both of the above.

This can all be mixed up with the work mentioned above on more aggressive proxying / barring of inter-tranche references for all non-proxied objects.

[sageserpent-open]

Another commit 1164e5679e53b4f051c6ffd2201573fee2fe9320, this one is with aggressive proxying, but with inter-tranche references enabled for non-proxied objects, essentially the same as c7836ff82ef5b6fcfc8bb9d1628af79f6c6a94e1:

Not great.

[sageserpent-open]

Tried option 1 from above and combined this with barring inter-tranche references, plus some cleanup of the messy hacks that were put into 'ImmutableObjectStorage'. As of commit dcbb070ef287f75e3553801437c067be97f6203e, what we have is:

[sageserpent-open]

That is 117 milliseconds per revision averaged over the section of the curve shown with a linear fit. Some progress, then.

It is now the case that delayed proxy loading is now confined to just queries, and that this is down to levels of structure in the hash maps in 'BlobStorageInMemory' being fetched bit by bit, which is as intended. I haven't seen any evidence of internal structure of the 'Vector' instances stored within each hash map being proxied, which would certainly contribute to an overall quadratic dependency, and with a low coefficient due to the fact that only one or two lifecycles are revised in each revision by the benchmark. Let's investigate ...

[sageserpent-open]

From inspection of the implementation of 'Vector', there are two problems regarding structure sharing using proxies. The first is simply that a vector instance uses plain old arrays internally, and these cannot be proxied by virtue of being final and being subclasses of 'AnyRef' and not extending any interface that a proxy could be built on. The second is that the new vector instances in a blob storage object are built with a call to 'patch', which uses the default implementation of using a builder to copy sections in from the patched vector, one element at a time - so there is no shared structure anyway.

This has given linear behaviour in the absence of persistence because each particular lifecycle in a blob storage only gets sporadic updates up to a fixed number - the events are darting around a set of item ids, whose size is proportional to the overall number of steps. So overall, the vectors are bounded in size as the benchmark runs, and thus any quadratic behavior is suppressed at large scales.

This is OK, as in the real world we expect an id to have a maximum number of events that will refer to it before it is annihilated.

On the other hand, when persistence is involved, all queries have to fetch blob storage objects from a tranche once the cache is full, jumping across all the ids - so every query will have to deal with vectors whose average size gets larger and larger as the benchmark runs - hence the quadratic behavior at large scale.

[sageserpent-open]

What to do?

If there was some abstraction like 'Vector', in that it allowed random access by index, but also uses structure sharing internally that could be proxied, and avoided the use of builders, then that's the obvious way to go - replace 'Vector' in 'BlobStorageInMemory' with the new wonder collection.

Failing that, the option mentioned above where a 'BlobStorage' has its own storage mechanism looks very tempting.

[sageserpent-open]

As an experiment, inter-tranche references for non-proxied objects were added back in on top of commit dcbb070, the idea being to see if a vector could be pulled incrementally out of storage, using the cache to hold bits of it in memory.

[sageserpent-open]

The contrasting full curve for dcbb070 is:

[sageserpent-open]

Clearly, the policy of forbidding non-proxy inter-tranche references works better. Perhaps there is room for making some exemptions to pick up the likes of 'Vector'.

[sageserpent-open]

I've tried reimplementing 'BlobStorageInMemory' in terms of 'TreeMap' rather than 'Vector' for the collection type underlying the lifecycles. This passes the tests as of a8a39802c7d1e252378f403d3fcd3e37181669f8, but unfortunately the code in the red-black tree implementation used by 'TreeMap' (which is the obvious candidate for structure sharing) has been written to use inlined fields that aren't proxied correctly. So once again the Scala collections library has resisted attempts to introduce proxies.

Hmm.

Having to write a custom implementation of the 'SortedMap' trait seems like overkill - time to consider a separate storage mechanism for blobs.

[sageserpent-open]

The latest is commit fea80406e87873d65e904ac5fb195a978ae59ab4 - this uses yet another implementation of 'BlobStorageinMemory' that uses a finger tree as a backend for the lifecycles, which in theory at least should permit structure sharing. The results still show a quadratic dependency, albeit one with a small coefficient:

[sageserpent-open]

It turned out that the internals of a finger tree were not being shared. Commit 211caa80eef6ec5a284c017a25e1bff05f622b0a fixes that successfully, leading to the following curves:

Note that there is still a quadratic dependency.

[sageserpent-open]

While the structure sharing issue has been sorted out within a blob storage, there is another potential cause of the quadratic dependency. As the benchmark jumps around its pool of item ids, although for each event it works only with two items, it will construct histories that involve more and more items chained together by object references dependencies.

As rendering an item will have to pull out the blob storage snaphots for all the related items, this will force an increasing number of tranches to be loaded as rendering follows the inter-item references for each new revision.

As an experiment, in commit: bdacca2bcdcd2b4acb2ee3a2a1dd5778376cc2be the blob storage objects have been kept in memory and are not stored via 'ImmutableObjectStorage', whereas everything else in 'Timeline' is. The results are gratifying, these include queries:

[sageserpent-open]

So we have 33 milliseconds for a revision and a query combined, this includes storing the rest of a timeline on to H2, although the blob storage isn't in this case.

All of this points to storing blob storage data via a separate mechanism, or at least splitting the storage so that timelines and blob storage objects are stored separately - that would allow the implementation in commit bdacca2bcdcd2b4acb2ee3a2a1dd5778376cc2be to use its cache of blob storage objects to yield linear performance, while reusing 'ImmutableObjectStorage'.

[sageserpent-open]

Another approach is to cutover the item proxies to perform lazy loading of related items, and stick with the current scheme of persisting the entire timeline via 'ImmutableObjectStorage', blob storage included... or do both, even.

[sageserpent-open]

Regarding the comment above about inter-tranche references, examination of the transitive closure of related items in the benchmark shows that the closure size remains low, and for may queries there simply isn't an item with the query item id for the scope used in the query.

So that theory is disproved.

The current line of investigation is to see what happens if blob storage objects are written to H2 as usual, but tranches are never purged from memory. Obviously this won't scale, but it may provide some more insight...

[sageserpent-open]

So, modifying commit 612dbfd55e7263fd0a5f2e08ed74cf582762cc1e so that tranches are not evicted from the cache yields the following graph:

Bear in mind that tranches are still being written for blob storage objects too.

Note the quadratic dependency is still there. (NB: this curve was obtained from a different computer than the other curves, so don't read anything into the coefficients).

[sageserpent-open]

Taking the code used to generate the previous curve and swapping H2 for a fake tranche implementation also yields a curve with quadratic behavior, albeit with much greater garbage collection pauses:

[sageserpent-open]

As of commit 0924d781d084b20fd5ba4856b3a38a5b7b2f59a6, corrected a performance regression that had crept into the code when it was changed to support only inter-tranche references by proxies.

This yields a speed up of 10x over 211caa80eef6ec5a284c017a25e1bff05f622b0a, but the quadratic dependency is still there:

[sageserpent-open]

Sanity check - did the recent changes to 'BlobStorageInMemory' introduce yet another quadratic dependency? Here are the results from running with 'WorldEfficientInMemoryImplementation' as of commit 0924d781d084b20fd5ba4856b3a38a5b7b2f59a6:

Very nice and linear, note that queries are included.

The gaps are due to the quick-and-dirty way the benchmark times are harvested, this highlights the pauses due to garbage collection. Using Scalameter doesn't show these gaps.

[sageserpent-open]

Ok, so maybe things have changed recently that have affected revising rather than querying - let's try again with the code from commit 0924d781d084b20fd5ba4856b3a38a5b7b2f59a6 modified to not make queries:

Back to being linear again, so queries and the persistent implementation are the culprit, none of the shared code is the cause.

[sageserpent-open]

Adding some logging to the code from commit 0924d781d084b20fd5ba4856b3a38a5b7b2f59a6 yields the following trace:

****** REVISING ******
Stored payload of size: 9656 for top level object: 1210552491 of type: class com.sageserpent.plutonium.AllEventsImplementation
Stored payload of size: 2232 for top level object: 813557272 of type: class quiver.Graph
Stored payload of size: 5024 for top level object: 507561779 of type: class com.sageserpent.plutonium.BlobStorageInMemory
@@@@@@ QUERYING @@@@@@
Loaded payload of size: 10337 for top level object: 1495794330 of type: class com.sageserpent.plutonium.AllEventsImplementation
Loaded payload of size: 2136 for top level object: 1583799598 of type: class quiver.Graph
Loaded payload of size: 5427 for top level object: 1970198177 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Loaded payload of size: 4854 for top level object: 1344020873 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 251358 to bring in: 169699925 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 4559 for top level object: 894909320 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 248511 to bring in: 1066109672 of type: class scala.collection.immutable.HashMap$HashMap1
Loaded payload of size: 4877 for top level object: 737048298 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 244890 to bring in: 2101744796 of type: class de.sciss.fingertree.FingerTree$Deep
Loaded payload of size: 4914 for top level object: 1555744607 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 216105 to bring in: 894127975 of type: class de.sciss.fingertree.FingerTree$Three
Using tranche: 216105 to bring in: 608737473 of type: class de.sciss.fingertree.FingerTree$Single
Loaded payload of size: 5244 for top level object: 753716054 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 150744 to bring in: 371280632 of type: class de.sciss.fingertree.FingerTree$Three
Loaded payload of size: 5327 for top level object: 1621290538 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 106374 to bring in: 1512311274 of type: class scala.collection.immutable.$colon$colon
Loaded payload of size: 5554 for top level object: 1314853217 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 251121 to bring in: 296165409 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 4663 for top level object: 1564976913 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 243138 to bring in: 937985817 of type: class scala.collection.immutable.HashMap$HashTrieMap
Using tranche: 106374 to bring in: 271326179 of type: class scala.collection.immutable.$colon$colon
Loaded payload of size: 1450 for top level object: 165908732 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 251301 to bring in: 872918411 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 5691 for top level object: 261479266 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 249378 to bring in: 1578857599 of type: class scala.collection.immutable.HashMap$HashTrieMap
Using tranche: 117243 to bring in: 1982772406 of type: class scala.collection.immutable.HashMap$HashMap1
Loaded payload of size: 4341 for top level object: 811587236 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 54459 to bring in: 1848483917 of type: class scala.collection.immutable.$colon$colon
Loaded payload of size: 4717 for top level object: 793345563 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 251400 to bring in: 1051353581 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 5323 for top level object: 167804561 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 250821 to bring in: 1419138042 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 4783 for top level object: 2037310239 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 209172 to bring in: 1042821209 of type: class scala.collection.immutable.HashMap$HashTrieMap
Using tranche: 72585 to bring in: 727338791 of type: class scala.collection.immutable.HashMap$HashMap1
Using tranche: 106374 to bring in: 678616645 of type: class de.sciss.fingertree.FingerTree$Three
Using tranche: 54459 to bring in: 1104042559 of type: class scala.collection.immutable.$colon$colon
Loaded payload of size: 5597 for top level object: 882506928 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 251142 to bring in: 1071910180 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 4874 for top level object: 1031450121 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 246171 to bring in: 1207954953 of type: class scala.collection.immutable.HashMap$HashTrieMap
Using tranche: 72585 to bring in: 1369514495 of type: class scala.collection.immutable.HashMap$HashMap1
Loaded payload of size: 4590 for top level object: 1078001993 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 53016 to bring in: 2104418239 of type: class scala.collection.immutable.$colon$colon
Loaded payload of size: 5563 for top level object: 914556765 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 251406 to bring in: 1754936733 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 1471 for top level object: 131748144 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 251331 to bring in: 2058734852 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 4976 for top level object: 1925281994 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 240441 to bring in: 1038685788 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 4500 for top level object: 1394981221 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 71475 to bring in: 805007511 of type: class scala.collection.immutable.HashMap$HashMap1
Using tranche: 53016 to bring in: 849534678 of type: class scala.collection.immutable.$colon$colon
Using tranche: 251406 to bring in: 1236577798 of type: class scala.collection.immutable.HashMap$HashTrieMap
Using tranche: 251289 to bring in: 1871566665 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 4678 for top level object: 369696612 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 227859 to bring in: 1583027307 of type: class scala.collection.immutable.HashMap$HashTrieMap
Using tranche: 71475 to bring in: 1913501199 of type: class scala.collection.immutable.HashMap$HashMap1
Loaded payload of size: 5002 for top level object: 304998266 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 38214 to bring in: 2026388396 of type: class scala.collection.immutable.$colon$colon
Loaded payload of size: 4748 for top level object: 1815369494 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 251379 to bring in: 932341990 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 5644 for top level object: 1435149103 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 250353 to bring in: 1655643127 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 5129 for top level object: 952553356 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 237798 to bring in: 1789583518 of type: class scala.collection.immutable.HashMap$HashTrieMap
Using tranche: 38214 to bring in: 1276126327 of type: class scala.collection.immutable.$colon$colon
Using tranche: 251406 to bring in: 1199014709 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 4879 for top level object: 2067771168 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 251337 to bring in: 568480027 of type: class scala.collection.immutable.HashMap$HashTrieMap
Using tranche: 226974 to bring in: 2143677705 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 4891 for top level object: 177343048 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 41364 to bring in: 1391298110 of type: class scala.collection.immutable.HashMap$HashMap1
Loaded payload of size: 4893 for top level object: 1922212907 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 37395 to bring in: 1459475259 of type: class scala.collection.immutable.$colon$colon
Loaded payload of size: 4946 for top level object: 194615570 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 251280 to bring in: 1100482671 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 5010 for top level object: 972605393 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 45153 to bring in: 1704072679 of type: class scala.collection.immutable.HashMap$HashMap1
Using tranche: 37395 to bring in: 242348372 of type: class scala.collection.immutable.$colon$colon
Loaded payload of size: 5376 for top level object: 493270557 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 250395 to bring in: 1741801345 of type: class scala.collection.immutable.HashMap$HashTrieMap
Using tranche: 241011 to bring in: 1630041994 of type: class scala.collection.immutable.HashMap$HashTrieMap
Using tranche: 45153 to bring in: 870588744 of type: class scala.collection.immutable.HashMap$HashMap1
Loaded payload of size: 5548 for top level object: 1563019552 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 33147 to bring in: 971765947 of type: class scala.collection.immutable.$colon$colon
Using tranche: 251400 to bring in: 458595223 of type: class scala.collection.immutable.HashMap$HashTrieMap
Using tranche: 33147 to bring in: 1733485957 of type: class scala.collection.immutable.$colon$colon
Loaded payload of size: 4782 for top level object: 1669494792 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 250518 to bring in: 1233570240 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 5114 for top level object: 1742504990 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 249873 to bring in: 1318516316 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 1304 for top level object: 1444135845 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 67305 to bring in: 1400907053 of type: class scala.collection.immutable.HashMap$HashMap1
Loaded payload of size: 5150 for top level object: 767351673 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 251388 to bring in: 1612457062 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 5315 for top level object: 1109311976 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 217074 to bring in: 363196627 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 4401 for top level object: 488566913 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 251385 to bring in: 1249729501 of type: class scala.collection.immutable.HashMap$HashTrieMap
Using tranche: 250995 to bring in: 721747487 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 5621 for top level object: 1870209348 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 212682 to bring in: 2011365254 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 1471 for top level object: 972233797 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 251367 to bring in: 1049387746 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 4540 for top level object: 1134744541 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 250692 to bring in: 459183820 of type: class scala.collection.immutable.HashMap$HashTrieMap
Using tranche: 248523 to bring in: 1503040227 of type: class scala.collection.immutable.HashMap$HashTrieMap
Loaded payload of size: 1507 for top level object: 1159387950 of type: class com.sageserpent.plutonium.BlobStorageInMemory
Using tranche: 243636 to bring in: 346512595 of type: class scala.collection.immutable.HashMap$HashTrieMap

Histogram:

Class: class de.sciss.fingertree.FingerTree$Deep, count: 1
Class: class scala.collection.immutable.$colon$colon, count: 12
Class: class de.sciss.fingertree.FingerTree$Single, count: 1
Class: class scala.collection.immutable.HashMap$HashTrieMap, count: 37
Class: class scala.collection.immutable.HashMap$HashMap1, count: 10
Class: class de.sciss.fingertree.FingerTree$Three, count: 3

[sageserpent-open]

Looks like the hash maps used in 'BlobStorageInMemory' are indeed exhibiting structure sharing, but nevertheless exhibit long chains of tranche fetches. The numbers for the other classes involved tend to stay low throughout the run.

Which hash map is it, the one to look up lifecycles for an item id, or the one to look up the revision for a recording id, or both?

[sageserpent-open]

Right - I stated before that 'BlobStorage' needs to have its own storage mechanism - after all, the clue is in the name.

It is clear that everything else plays nicely with 'ImmutableObjectStorage' and would give linear scaling; furthermore, by breaking 'BlobStorage' off from 'ImmutableObjectStorage', the rather messy splitting of a timeline into three subpieces in commit 025119ec9c08b84e9cc10ddf71c824caa974be99 could perhaps be reverted.

Another benefit would be that the existing escape-hatch hack allowing a blob storage to be extracted out of a session so that a scope can use it could be closed off, as a scope could then make a blob storage directly from a database connection - no session is required.

There is the business of coordinating updates to the database from both 'ImmutableObjectStorage' and whatever will underlie 'BlobStorageOnH2' (to give it a name), but that falls under the area of transaction support, which isn't in scope for this issue.

One thing - the existing tests for 'BlobStorageInMemory' won't work with 'BlobStorageOnH2' - the latter will need to fix its type parameters to work effectively with H2. How about taking the correctness of 'BlobStorageInMemory' as a given, as it has good tests, and writing a comparison test between the two implementations? This is done for the competing world implementations already, so is a workable approach.

[sageserpent-open]

Oh - another thing, by having 'BlobStorageOnH2' go to H2 to satisfy queries, we avoid having to haul in loads of irrelevant data into memory just to satisfy a query for a single id. Just saying...

[sageserpent-open]

After a very long slog, there is finally an implementation of 'BlobStorageOnH2' that passes its test, see commit f20b8c67 - however it runs far too slowly for practical use, and is currently specialised to just test data types. Cue a fresh round of optimisation, refactoring, etc...

[sageserpent-open]

Ideas:

1. Consider using a final join with rather than a select from the common table expression DominantEntriesByItemIdAndItemClass so that payloads aren't extracted into DominantEntriesByItemIdAndItemClass when they aren't needed.

2. Store payloads in a separate table, as most of the query logic doesn't care about them anyway.

3. Break out the item id and item class into a separate table with associated Scala hashes of the item id and item class objects. Queries can use the Scala hashes and then filter through results to weed out extraneous items whose id or class hash has collided with the query.

4. Use some indices!

5. Supply the item id / exact item class to SQL queries when possible to thin out the massive joins.

[sageserpent-open]

Also need to address the hardwiring of simplified test types into 'BlobStorageOnH2'.

This can either be reverted completely - it was introduced in commits 04a72770 and ef42faa1, or it can be generalised so that 'BobStorageOnH2' has both test and production implementations. Doing this would allow 'BlobStorageSpec' to test both 'BlobStorageInMemory' and 'BlobStorageOnH2' in its test typed form, and 'BlobStorageOnH2Spec' can test 'BlobStorageOnH2' in its production typed form.

This can wait until the optimisation work is done, though.

[sageserpent-open]

As of commit caf3d49b3d218575071ee01dbf2640831c3f042e there is a rather hacked version of 'BlobStorageonH2' that despite using indices and some thinning of the query logic, performs dismally in 'BlobStorageOnH2Spec'.

It's still worth a quick experimental hack to wire it into 'WorldH2StorageImplementation' to see how it performs in the benchmark, not least because this needs doing whatever alternative blob storage implementation is chosen.

[sageserpent-open]

While this is going on, the Department of Crazy Ideas has a new proposal: way back in #16, an idea was floated to use immutable maps that were coupled to a persistent backend; this went by the wayside due to the notion of the imperative operations on the backend not fitting with the outwardly immutable map API.

Nowadays such fineries are out of the window: blob storage is more or less an immutable API, albeit using the builder pattern for making a revision from an existing immutable object; yet this is implemented in top of an imperative persistent backend, without any wrapping up in an effect monad or whatever.

Now that is has been seen to be possible to layer an immutable API on top of an SQL database, perhaps it is worth doing this for a map implementation - most of the pain in the current implementation of 'BlobStorageOnH2' is down to having to deal with time interactions with lineage revisions, as well as with unpicking the Scala tuple orderings in a manner that can be expressed in SQL.

If so, the plan would be to keep using 'ImmutableObjectStorage' as the default storage mechanism, but to introduce a pluggable form of 'BlobStorageInMemory' that uses allows tailoring of the what implements its internal maps, so they can use the new database-backed implementation.

Come to think of it, this could be done across the board for the various maps in 'AllEventsImplementation' and in the item state updates dag. Hmm....

[sageserpent-open]

Progress so far:

Going to increasing number of revisions past 300 000 has revealed more scaling gotchas in addition to the problem with queries via the blob storage implementation discussed above:

1. The Scala library HashMap doesn't play well with 'ImmutableObjectStorage'. This was already known in the context of 'BlobStorageInMemory', but has also become a problem elsewhere in 'Timeline' due to its use in underpinning the item state updates dag and the lifecycles buried within 'AllItemsImplementation'. A quick-and-dirty replacement, 'ScuzzyMap', has been substituted and experiments are underway to verify whether this completely solves this new scaling issue, so far the results are encouraging, but not conclusive due to the query issue.

2. H2 has an 'MVStore' backend storage implementation that is exhibiting livelock once the benchmark has gone well beyond 100 000 steps. The livelock kicks in periodically and lasts for several minutes, eventually yielding to normal performance for another 10 minutes or so, then kicking back in again. This may be connected with the analysis tasks that H2 automatically schedules to optimise the use of indices. As this optimisation is desirable, the workaround has been to use the older 'PageStore' backend for now, but 'MVStore' is the one to go for long term.

Right now, results are in for commit 623aedb0a621bd3acc77d2342c2d1f475ad99a9d, these still show some quadratic behaviour:

[sageserpent-open]

The next step is to disable querying to see if this is still the source of this quadratic dependency.

If it is, then perhaps it is worth reinstating 'BlobStorageInMemory', now that 'ScuzzyMap' has taken over from 'HashMap'. Another possibility is to use Redis or some other backend storage technology instead of H2 (or any other SQL database), as the SQL queries issued by 'BlobStorageOnH2' are fairly ham-fisted and only allow limited optimisation via table indices.

[sageserpent-open]

If queries are still the only source of quadratic scaling, then it would be useful to experiment with switching back to 'HashMap' elsewhere to see precisely what impact it had, so far this has been mixed up with other scaling problems.

[sageserpent-open]

With queries disabled from commit 623aedb0a621bd3acc77d2342c2d1f475ad99a9d:

[sageserpent-open]

So again, there is roughly the same quadratic coefficient, which would seem to imply that 'BlobStorageOnH2' is indeed roughly linear after all.

So has the quadratic behavior been reintroduced?

[sageserpent-open]

Results from 0924d781d084b20fd5ba4856b3a38a5b7b2f59a6 with querying disabled, taken over more steps...

[sageserpent-open]

Ok, so let's review 0924d781d084b20fd5ba4856b3a38a5b7b2f59a6, but with queries disabled - this was discussed way back above on May 11th 2019 when it started to look like querying via 'BlobStorageInMemory' was the cause of the quadratic scaling.

Here are the results from another run with more steps, the graph does not show the initial warm-up up to step 250 000, as there is some irregularity.

Almost linear. By contrast, fitting a quadratic to the original graph from May 11th has a quadratic coefficient of 5e-6 and a linear one of 9.53. Similar magnitudes, then.

[sageserpent-open]

Back to the present, we have commit 623aedb0a621bd3acc77d2342c2d1f475ad99a9d, removing queries and again focussing on steps past the warm up from 250 000 onwards yields this:

Bear in mind that this is using 'BlobStorageOnH2' to hit H2 when doing revisions, both for data reads and for writes, even though the benchmark has had queries disabled.

[sageserpent-open]

So now we're looking past step 250 000, what happens if we review the graph for commit 623aedb0a621bd3acc77d2342c2d1f475ad99a9d with queries included?

[sageserpent-open]

There is a small quadratic (or is it N * Log(N)) dependency with quadratic dependency of ~8e-5 with commit 623aedb0a621bd3acc77d2342c2d1f475ad99a9d over a long series of steps, contrast this with the 'definitely linear' commit 0924d781 with queries disabled whose quadratic coefficient is ~1e-5.

I think more data from more steps is required for both examples.

I also wonder whether there is merit in dusting off a 'ganged' blob storage that writes to both an in-memory data structure and H2, but supports reads via in-memory for revisions and H2 for queries.

[sageserpent-open]

So, another look at 0924d78 over more steps, with no queries and using the old H2 page store:

Essentially the same story, but at least there are no surprises going up to 1 000 000 steps. The linear coefficient is significantly larger, presumably this is due to the H2 page store being less performant than the later MVStore.

[sageserpent-open]

One thing - just prior to hitting the 1 000 000 step mark, performance went haywire, this was omitted in the previous graph to allow a curve fit. Here's the whole thing:

[sageserpent-open]

Here is 623aedb, with queries included over more steps:

[sageserpent-open]

Here is the latest iteration of using 'BlobStorageOnH2', the use of 'HashMap' has been reinstated, but H2's page store is still in use. Caching on H2 is back down to the default, extra indices have been aded and auto-analysis is in use. Commit f0685021be36ba32867f13401ffb387c053e2372:

[sageserpent-open]

In summary so far (including some results that are not shown):

1. 'ScuzzyMap' didn't deliver any benefit in the context of using 'BlobStorageOnH2', but what wasn't tried, at least with the latest set of changes. was to use 'BlobStorageInMemory' across the board in combination with 'ScuzzyMap'.

2. It may be worth giving H2 some more cache memory, but this has no bearing on the quadratic scaling.

3. Right now, it is clear that 'BlobStorageOnH2' definitely requires H2 to use the page store implementation. This may turn out to be down to the rather naive lack of coordination between 'ImmutableObjectStorage' using H2 and 'BlobStorageOnH2' - I wonder if delayed writes from one are causing livelock on the other, to be honest the whole business of how the database is driven needs to be overhauled, right now there is a rather ham-fisted use of the synchronous ScalikeJDBC API driving H2 which is in turn performing asynchronous disk writes under the hood.

4. I think that 'BlobStorageOnH2' is the major source of the quadratic scaling (or is it N * log(N) ?). Given how long it takes to get results, I'm taking the view for now that while it was a good idea, did show successful use of indices in a database and that a separate storage mechanism is feasible, it is not the way forward for a system that has to work with increasingly larger amounts of stored data. Perhaps a different flavour of SQL database might work better, but my feeling is that the SQL queries emitted by 'BlobStorageOnH2' are themselves problematic.

5. For now, rather than carrying on with 'BlobStorageOnH2', how about using the same approach of a separate storage implementation for the 'BlobStorageAPI', but using, say, Redis? I am tempted to apply this for the tranches implementation backing 'ImmutableObjectStorage' as well, but combinations could be tried to see how the scaling is affected. NO BENEFIT IN USING REDIS FOR TRANCHES

6. Based on 'BlobStorageOnH2', it is also feasible to have an implementation of the Scala immutable map API sitting on top of H2 - and the SQL queries for this would be a lot simpler than for 'BlobStorageOnH2', perhaps to the point of scaling well. So another approach would be to go back to in-memory data structures across the board using 'ImmutableObjectStorage' and H2, but to use this special H2-backed map implementation, in rather the same way as 'ScuzzyMap'.

7. Let's not forget what what stated in point 1 above - 'ScuzzyMap' could also be used with a purely in-memory implementation. ABANDONED - VERY POOR SCALING.

So, lots of ideas, and yet another round of benchmarking...

[sageserpent-open]

So, regarding point 7 from the comment above, here is commit a57c9228eef3f16cbb53dc33f2921a51369d46d7 modified to use 'BlobStorageInMemory'. 'ScuzzyMap' is in use here across the board, and queries are included:

This is clearly not a workable approach.

[sageserpent-open]

Commit 77c92e0f38dac4b69c3a8ee212ae00fd34067b63: use an additional table broken out from 'Snapshot' to calculate dominant revisions in the implemenatation of 'BlobStorageOnH2', and removed what are now clearly unused indices. The other two indices are definitely being used now.

[sageserpent-open]

Commit eb31b2ec07bdfadfbf9b2bed436a4df4a9d2a513: shadowing the BlobStorageOnH2 with the in-memory implementation so that revisions don't have to go to H2.

[sageserpent-open]

Commit feaa3e97b3e085e71b47dd4a64ee448dcac4e377: the point here is to show the change to 'BlobStorageonH2' where there is an extra table used to calculate dominant revisions.