search

OSGeo / PROJ

PROJ - Cartographic Projections and Coordinate Transformations Library
https://proj.org
Other
1.72k stars 780 forks source link

ERROR 1: PROJ: Cannot take exclusive lock on .local/share/proj/cache.db #3959

Open MarcYin opened 10 months ago

MarcYin commented 10 months ago

When import gdal, the PROJ library throws error:

ERROR 1: PROJ: Cannot take exclusive lock on .local/share/proj/cache.db

This only happens when I have multiple computing nodes using the same conda environment

from osgeo import gdal

Problem description

Expected Output

Environment Information

Installation method

rouault commented 10 months ago

@MarcYin It would be useful if you could provide a minimum Python reproducer with either gdal or pyproj4

You can also try setting the PROJ_LOCK_MAX_ITERS environment variable to a value greater than 30. 30 corresponds to ~ 1 second of retry attempts to get the exclusive lock

MarcYin commented 10 months ago

Hi,

It is a bit python code involves creating VRT from list of tif file to resample them to 10 meters and read:

from osgeo import gdal

url_header = '/vsicurl/https://gws-access.jasmin.ac.uk/public/nceo_isp/S2/30/U/VB/S2A_MSIL1C_20211028T111201_N0301_R137_T30UVB_20211028T131717.SAFE/GRANULE/L1C_T30UVB_A033167_20211028T111452/IMG_DATA/T30UVB_20211028T111201_'
sur_refs = [f'{url_header}{band}_sur.tif' for band in ['B02', 'B03', 'B04', 'B05', 'B06', 'B07', 'B08', 'B8A', 'B11', 'B12']]

# Reflectance
g = gdal.BuildVRT('', sur_refs, separate=True, xRes=10, yRes=10, resampleAlg=gdal.GRIORA_NearestNeighbour)
data = g.ReadAsArray()
print(data.shape)
rouault commented 10 months ago

It is a bit python code involves creating VRT from list of tif file to resample them to 10 meters and read:

that doesn't hit the bug. I guess you need to add some multiprocessing/multithreading on top of that to build a full reproducer

MarcYin commented 10 months ago

Hi,

I was trying to do that but it seems not tigger the error. But I hope a bit more context on this will help to identify the potential issue. The processing involves:

  1. reading the Sentinel-2 data using the code I posted above (potential issue here for crossing meridian line?)
  2. reading a land cover map in British National Grid projection covering the Sentinel-2 tile with gdalwarp (potential issue here for projecting the British National Grid to the different MGRS tile projection?)

The Sentinel-2 data covers the entire UK, and the reading of data is submitted to different computing nodes sharing the same conda environment.

The error I mentioned above did not trigger all the time, but only occasionally.

Thanks.