search

smsearcy / mesh-info

Collect and view information about an AREDN mesh network.
GNU General Public License v3.0
5 stars 2 forks source link

Optimize saving nodes and/or links to database #116

Open smsearcy opened 1 year ago

smsearcy commented 1 year ago

The database updates are taking 5-10 seconds, which is too long (particularly since I've seen a couple of SQLite locking issues recently, which is going to stem from writes locking the database).

Most likely the culprit is the repeated queries to look for the appropriate node/link to update. It is often faster to minimize the number of queries, by preloading more data.

Given the complexity of matching the node, I think the easiest win (and possibly adequate) is to start with optimizing the saving of links.

smsearcy commented 5 months ago

Another option is leveraging the bulk update features in SQLAlchemy (although it's likely the queries in a hot loop are more to blame).

https://docs.sqlalchemy.org/en/20/orm/queryguide/dml.html#orm-bulk-update-by-primary-key