search

peterwebster / henson

Master data store for the Hensley Henson Journals project, and issue tracker. The application code is kept elsewhere.
1 stars 1 forks source link

Review performance of data model as data volumes have increased #250

Open peterwebster opened 4 years ago

peterwebster commented 4 years ago

From @ojlyytinen for future attention:

"One possible thing to investigate, if you get Gregory to do some work at some point, would be to see how the performance of the data model is scaling with more material going in. Fedora scales really poorly when it comes to relations between records. Even a hundred or so related records can slow updates significantly. I'm not sure exactly how your data is stored in Fedora so you may or may not be affected by this. But it would be something that just progressively gets slower as more material gets put in. So if you're seeing that ingest is just constantly getting slower and slower regardless of server load then this explanation would fit that pattern.

At a quick look, I can see that for example Isabella Henson is related to 465 journal entries. Just fully loading up this record in console takes a whopping 40 seconds.

There are ways to somewhat get around this scaling problem by being very careful about only loading the parts that are strictly needed or trying to reuse already loaded records. And in some cases where we expect lots of related records we have even changed the way they are stored in Fedora to speed it up further.

peterwebster commented 2 years ago

Note the load time for Ella Henson (with over 1,000 related entries) at https://henson.durham.ac.uk/people/person112

But, for bishop Gore (c200 entries) it is slightly slow, but probably acceptable so. https://henson.durham.ac.uk/people/person49