pyproj4 / pyproj

Python interface to PROJ (cartographic projections and coordinate transformations library)
https://pyproj4.github.io/pyproj
MIT License
1.05k stars 212 forks source link

PERF: drop GIL during long-running proj database calls #1354

Closed tpwrules closed 1 year ago

tpwrules commented 1 year ago

These calls can take hundreds of milliseconds to access the PROJ databases, during which time no other Python code in any other thread can run. Use nogil to avoid this effect.

This should be safe for these calls as they all take a separate PROJ threading context.

It's quite probable there are other calls which could use this, but these negatively affected my application and I was able to identify them. I would welcome additions to the set of calls to wrap.

codecov[bot] commented 1 year ago

Codecov Report

Merging #1354 (42153ea) into main (3cea713) will not change coverage. The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1354   +/-   ##
=======================================
  Coverage   96.41%   96.41%           
=======================================
  Files          20       20           
  Lines        1812     1812           
=======================================
  Hits         1747     1747           
  Misses         65       65           
tpwrules commented 1 year ago

It doesn't improve execution time, just makes other threads in the process more stable. It's a really hard thing to benchmark and measure unfortunately. I just was able to observe the effect on my program, but I can't really share the program here.

jjimenezshaw commented 1 year ago

I don't know enough about the thread-safety of PROJ

PROJ does not deal with thread safety internally. It should be managed by the user via the "context". The same context must not be used simultaneously by two different threads. (PROJ uses the "context" to keep there caches and any global initialization. So new context, new initialization. The access to the sqlite db -proj.db- is in read only mode, and managed by sqlite3 to avoid collisions)

snowman2 commented 1 year ago

Thanks @tpwrules 👍

snowman2 commented 1 year ago

I would be interested to see if doing this with proj_create calls helps as well.