More efficient query (faster) where multiple location IDs scope the ID query

WildMeOrg / Wildbook

Wild Me's first product, Wildbook supports researchers by allowing collaboration across the globe and automation of photo ID matching

https://www.wildme.org/wildbook.html

GNU General Public License v2.0

103 stars 64 forks source link

More efficient query (faster) where multiple location IDs scope the ID query #667

Closed holmbergius closed 3 weeks ago

holmbergius commented 1 month ago

Where >1 location ID scopes an ID query (e.g., (enc.locationID=='place1') || enc.locationID=='place2' )), query completion time rises dramatically (e.g., 1 sec. -> 180 seconds). due to inefficient JDOQL -> SQL translation by DataNucelus. Changing to "literal" scoping produces approx. 100-1000x improvement in query execution time by reducing query complexity and leveraging in memory comparison.

Changes

ID query building now leverages literals for location ID

naknomum commented 1 month ago

whoa what?? this seems like some kind of dn/sql magic. amazing.