search

CenterForDigitalHumanities / TPEN-Newberry-Italian

T-PEN Newberry Italian paleography project
0 stars 0 forks source link

functions in parsing takes along time #63

Open Domhnall101 opened 7 years ago

Domhnall101 commented 7 years ago

add line and other functions take a while to happen. Make a line is so long to happen It is going to be a problem for the user hoping server upgrades will resolve the issue

thehabes commented 7 years ago

The theory and guts behind this may mean this delay sticks around, regardless of how good of a server we put this on. Here is why:

Mongo as a database solution queues up calls and completes them sequentially only once the call previous in the sequence has completed. There is no multi-threading, like with SQL.

The annotationStore API does not take advantage of Mongo subdocuments. This means when we make a change to a line, we also have to update that lines' properties in whatever annotationList it exist in, so multiple calls have to made per modular action, which queue up and complete at the speed Mongo can. This is where we feel the bottleneck, and unfortunately queuing these calls is the cause, not the speed of internet or server (although if you ended up with a painfully slow internet connection or server, you may experience that lag is greater than the Mongo lag. This is not the case for us, I believe we are experiencing lags at the speed of Mongo.) We did not optimize these pieces of functionality within the annotationStore API for performance, and moving to a better server will most likely not improve the Mongo lag.

The amount of work that would be needed to make the annotationStore more performant is outside the scope of this project. Be prepared for this delay to still exist when we move it to the new server, at which point we can further discuss what is needed to improve it.

thehabes commented 7 years ago

I was able to unwrap a $.post("getAnnos") from most of the create/delete/update functions, which has helped with multiple race condition scenarios and improved this just a bit. I track a lot of changes in memory instead of on the server.

thehabes commented 7 years ago

Although not fully optimized, I did some optimizing that was in scope. There should be a noticeable improvement in some places, and no change in others.