osm-search / Nominatim-Data-Analyser

QA Tool for Nominatim. Helps to improve the OpenStreetMap data quality and therefore the Nominatim search results.
GNU General Public License v2.0
10 stars 3 forks source link

Add multi threading feature #6

Open AntoJvlt opened 3 years ago

AntoJvlt commented 3 years ago

Adding multi threading to the tool would reduce a lot the time needed to execute all the rules. As python threads run concurrently the real benefit will come when query are executed on the PostgreSQL server (maybe by using a connection pool) and when we call clustering-vt. Clustering-vt and PostgreSQL queries are the most time consuming operations when executing a rule so threads would definitively improve the performance of the tool.

It can be interesting to check if it would be better to execute a whole rule in its own thread or if threads would be spawn locally when executing PostgreSQL queries and when calling clustering-vt.

In the first case, we should make sure some operations are thread safe, for example when accessing the config or when writing/reading to files.