There are three remaining areas that will speed up fetches server side
1) when the client requests a fetch it feeds all currently changed records that have not yet been persisted. The way this is done currently creates many additional "changed" records, where the change is actually null. This causes the server to prefetch these records. Some experiments show that cleaning this up on the client side before the fetch will result in about a 20% speedup on the server side.
2) the server at the end of the fetch reformats the gathered data into a json graph for the client. This should be completely redone, so that as each vector is fetched it is inserted into the json graph in its final format. This should save about 5-10%
3) related to #2 is the way the vectors are during the fetch process. Currently each vector is placed into a hash (the vector is the key, and the array of records that vector fetches is the value) Instead of a single flat hash, the structure should be a tree of hashes, so that each key is the next step down the vector path. This will correspond to the json graph, but importantly it will allow a faster vector look up since at each hash level there will only be few unique keys, and changes will occur most frequently towards the leafs.
There are three remaining areas that will speed up fetches server side 1) when the client requests a fetch it feeds all currently changed records that have not yet been persisted. The way this is done currently creates many additional "changed" records, where the change is actually null. This causes the server to prefetch these records. Some experiments show that cleaning this up on the client side before the fetch will result in about a 20% speedup on the server side. 2) the server at the end of the fetch reformats the gathered data into a json graph for the client. This should be completely redone, so that as each vector is fetched it is inserted into the json graph in its final format. This should save about 5-10% 3) related to #2 is the way the vectors are during the fetch process. Currently each vector is placed into a hash (the vector is the key, and the array of records that vector fetches is the value) Instead of a single flat hash, the structure should be a tree of hashes, so that each key is the next step down the vector path. This will correspond to the json graph, but importantly it will allow a faster vector look up since at each hash level there will only be few unique keys, and changes will occur most frequently towards the leafs.