search

OSGeo / gdal

GDAL is an open source MIT licensed translator library for raster and vector geospatial data formats.
https://gdal.org
Other
4.67k stars 2.47k forks source link

Check for `SQLITE3_MUTEX_ALLOC` fails randomly #10125

Open weiznich opened 1 month ago

weiznich commented 1 month ago

What is the bug?

I randomly get failures from cmake that complains that sqlite3_mutex_alloc is not defined, while it's actually defined in the used libsqlite3 library. We statically build this library with the same flags every time so there is really no way that this changes between builds.

CMake Build Output ``` CMAKE_TOOLCHAIN_FILE_x86_64-unknown-linux-gnu = None CMAKE_TOOLCHAIN_FILE_x86_64_unknown_linux_gnu = None HOST_CMAKE_TOOLCHAIN_FILE = None CMAKE_TOOLCHAIN_FILE = None CMAKE_GENERATOR_x86_64-unknown-linux-gnu = None CMAKE_GENERATOR_x86_64_unknown_linux_gnu = None HOST_CMAKE_GENERATOR = None CMAKE_GENERATOR = None cargo:warning=Compiler version doesn't include clang or GCC: "cc" "--version" cargo:warning=Compiler version doesn't include clang or GCC: "c++" "--version" CMAKE_PREFIX_PATH_x86_64-unknown-linux-gnu = None CMAKE_PREFIX_PATH_x86_64_unknown_linux_gnu = None HOST_CMAKE_PREFIX_PATH = None CMAKE_PREFIX_PATH = None CMAKE_x86_64-unknown-linux-gnu = None CMAKE_x86_64_unknown_linux_gnu = None HOST_CMAKE = None CMAKE = None running: cd "/builds/gstr/gst/target/test-jenkins/build/gdal-src-5f4a8daa0fa94874/out/build" && CMAKE_PREFIX_PATH="" "cmake" "/builds/gstr/gst/cargo/git/checkouts/gdal-5183a7959fbbd406/593a6b5/gdal-src/source" "-DGDAL_BUILD_OPTIONAL_DRIVERS=OFF" "-DOGR_BUILD_OPTIONAL_DRIVERS=OFF" "-DGDAL_USE_INTERNAL_LIBS=ON" "-DGDAL_USE_EXTERNAL_LIBS=OFF" "-DBUILD_SHARED_LIBS=OFF" "-DBUILD_STATIC_LIBS=ON" "-DBUILD_APPS=OFF" "-DBUILD_DOCS=OFF" "-DBUILD_TESTING=OFF" "-DBUILD_GMOCK=OFF" "-DPROJ_INCLUDE_DIR=//builds/gstr/gst/target/test-jenkins/build/proj-sys-e047e9baac18e3f9/out/include" "-DPROJ_LIBRARY=//builds/gstr/gst/target/test-jenkins/build/proj-sys-e047e9baac18e3f9/out/lib/libproj.a" "-DACCEPT_MISSING_LINUX_FS_HEADER=ON" "-DOGR_ENABLE_DRIVER_AVC=OFF" "-DOGR_ENABLE_DRIVER_CAD=OFF" "-DOGR_ENABLE_DRIVER_CSV=OFF" "-DOGR_ENABLE_DRIVER_DGN=OFF" "-DOGR_ENABLE_DRIVER_DXF=OFF" "-DOGR_ENABLE_DRIVER_EDIGEO=OFF" "-DOGR_ENABLE_DRIVER_FLATGEOBUF=OFF" "-DOGR_ENABLE_DRIVER_GEOCONCEPT=OFF" "-DOGR_ENABLE_DRIVER_GEOJSON=OFF" "-DOGR_ENABLE_DRIVER_GMT=OFF" "-DOGR_ENABLE_DRIVER_GTFS=OFF" "-DOGR_ENABLE_DRIVER_JSONFG=OFF" "-DOGR_ENABLE_DRIVER_MAPML=OFF" "-DOGR_ENABLE_DRIVER_OPENFILEGDB=OFF" "-DOGR_ENABLE_DRIVER_PGDUMP=OFF" "-DOGR_ENABLE_DRIVER_NTF=OFF" "-DOGR_ENABLE_DRIVER_S57=OFF" "-DOGR_ENABLE_DRIVER_SELAFIN=OFF" "-DOGR_ENABLE_DRIVER_SHAPE=ON" "-DOGR_ENABLE_DRIVER_SXF=OFF" "-DOGR_ENABLE_DRIVER_TAB=OFF" "-DOGR_ENABLE_DRIVER_TIGER=OFF" "-DOGR_ENABLE_DRIVER_VDV=OFF" "-DOGR_ENABLE_DRIVER_WASP=OFF" "-DOGR_ENABLE_DRIVER_IDRISI=OFF" "-DOGR_ENABLE_DRIVER_SDTS=OFF" "-DOGR_ENABLE_DRIVER_VRT=OFF" "-DOGR_ENABLE_DRIVER_MEM=OFF" "-DGDAL_ENABLE_DRIVER_AAIGRID=OFF" "-DGDAL_ENABLE_DRIVER_ADRG=OFF" "-DGDAL_ENABLE_DRIVER_AIGRID=OFF" "-DGDAL_ENABLE_DRIVER_AIRSAR=OFF" "-DGDAL_ENABLE_DRIVER_BLX=OFF" "-DGDAL_ENABLE_DRIVER_BMP=OFF" "-DGDAL_ENABLE_DRIVER_BSB=OFF" "-DGDAL_ENABLE_DRIVER_CALS=OFF" "-DGDAL_ENABLE_DRIVER_CEOS=OFF" "-DGDAL_ENABLE_DRIVER_COASP=OFF" "-DGDAL_ENABLE_DRIVER_COSAR=OFF" "-DGDAL_ENABLE_DRIVER_CTG=OFF" "-DGDAL_ENABLE_DRIVER_DIMAP=OFF" "-DGDAL_ENABLE_DRIVER_DTED=OFF" "-DGDAL_ENABLE_DRIVER_ELAS=OFF" "-DGDAL_ENABLE_DRIVER_ENVISAT=OFF" "-DGDAL_ENABLE_DRIVER_ERS=OFF" "-DGDAL_ENABLE_DRIVER_FIT=OFF" "-DGDAL_ENABLE_DRIVER_GFF=OFF" "-DGDAL_ENABLE_DRIVER_GIF=OFF" "-DGDAL_ENABLE_DRIVER_GRIB=OFF" "-DGDAL_ENABLE_DRIVER_GSG=OFF" "-DGDAL_ENABLE_DRIVER_GTIFF=OFF" "-DGDAL_ENABLE_DRIVER_GXF=OFF" "-DGDAL_ENABLE_DRIVER_HF2=OFF" "-DGDAL_ENABLE_DRIVER_HFA=OFF" "-DGDAL_ENABLE_DRIVER_ILWIS=OFF" "-DGDAL_ENABLE_DRIVER_IRIS=OFF" "-DGDAL_ENABLE_DRIVER_JAXAPALSAR=OFF" "-DGDAL_ENABLE_DRIVER_JDEM=OFF" "-DGDAL_ENABLE_DRIVER_JPEG=OFF" "-DGDAL_ENABLE_DRIVER_KMLSUPEROVERLAY=OFF" "-DGDAL_ENABLE_DRIVER_L1B=OFF" "-DGDAL_ENABLE_DRIVER_LEVELLER=OFF" "-DGDAL_ENABLE_DRIVER_MAP=OFF" "-DGDAL_ENABLE_DRIVER_MRF=OFF" "-DGDAL_ENABLE_DRIVER_MSGN=OFF" "-DGDAL_ENABLE_DRIVER_NGSGEOID=OFF" "-DGDAL_ENABLE_DRIVER_NIFT=OFF" "-DGDAL_ENABLE_DRIVER_NORTHWOOD=OFF" "-DGDAL_ENABLE_DRIVER_OZI=OFF" "-DGDAL_ENABLE_DRIVER_PCIDSK=OFF" "-DGDAL_ENABLE_DRIVER_PCRASTER=OFF" "-DGDAL_ENABLE_DRIVER_PNG=OFF" "-DGDAL_ENABLE_DRIVER_PRF=OFF" "-DGDAL_ENABLE_DRIVER_R=OFF" "-DGDAL_ENABLE_DRIVER_RAW=OFF" "-DGDAL_ENABLE_DRIVER_RIK=OFF" "-DGDAL_ENABLE_DRIVER_RMF=OFF" "-DGDAL_ENABLE_DRIVER_RS2=OFF" "-DGDAL_ENABLE_DRIVER_SAFE=OFF" "-DGDAL_ENABLE_DRIVER_SAGA=OFF" "-DGDAL_ENABLE_DRIVER_SAR_CEOS=OFF" "-DGDAL_ENABLE_DRIVER_SENTINEL2=OFF" "-DGDAL_ENABLE_DRIVER_SGI=OFF" "-DGDAL_ENABLE_DRIVER_SIGDEM=OFF" "-DGDAL_ENABLE_DRIVER_SRTMHGT=OFF" "-DGDAL_ENABLE_DRIVER_STACIT=OFF" "-DGDAL_ENABLE_DRIVER_STACTA=OFF" "-DGDAL_ENABLE_DRIVER_TERRAGEN=OFF" "-DGDAL_ENABLE_DRIVER_TGA=OFF" "-DGDAL_ENABLE_DRIVER_TIL=OFF" "-DGDAL_ENABLE_DRIVER_TSX=OFF" "-DGDAL_ENABLE_DRIVER_USGSDEM=OFF" "-DGDAL_ENABLE_DRIVER_XPM=OFF" "-DGDAL_ENABLE_DRIVER_XYZ=OFF" "-DGDAL_ENABLE_DRIVER_ZMAP=OFF" "-DGDAL_ENABLE_DRIVER_IDRISI=OFF" "-DGDAL_ENABLE_DRIVER_PDS=OFF" "-DGDAL_ENABLE_DRIVER_SDTS=OFF" "-DGDAL_ENABLE_DRIVER_VRT=OFF" "-DGDAL_ENABLE_DRIVER_MEM=OFF" "-DGDAL_USE_SQLITE3=ON" "-DSQLite3_INCLUDE_DIR=//builds/gstr/gst/cargo/registry/src/index.crates.io-6f17d22bba15001f/libsqlite3-sys-0.28.0/sqlite3" "-DSQLite3_LIBRARY=//builds/gstr/gst/target/test-jenkins/build/libsqlite3-sys-2a514ff13e459a59/out/libsqlite3.a" "-DOGR_ENABLE_DRIVER_SQLITE=ON" "-DOGR_ENABLE_DRIVER_GPKG=ON" "-DOGR_ENABLE_DRIVER_VFK=OFF" "-DGDAL_USE_HDF5=OFF" "-DGDAL_USE_NETCDF=OFF" "-DGDAL_USE_CURL=OFF" "-DOGR_ENABLE_DRIVER_AMIGOCLOUD=OFF" "-DOGR_ENABLE_DRIVER_CARTO=OFF" "-DOGR_ENABLE_DRIVER_DAAS=OFF" "-DOGR_ENABLE_DRIVER_EEDA=OFF" "-DOGR_ENABLE_DRIVER_ELASTIC=OFF" "-DOGR_ENABLE_DRIVER_NGW=OFF" "-DGDAL_ENABLE_DRIVER_OGCAPI=OFF" "-DGDAL_ENABLE_DRIVER_PLMOSAIC=OFF" "-DGDAL_ENABLE_DRIVER_WCS=OFF" "-DGDAL_ENABLE_DRIVER_WMS=OFF" "-DGDAL_ENABLE_DRIVER_WMTS=OFF" "-DGDAL_USE_POSTGRESQL=OFF" "-DGDAL_ENABLE_DRIVER_POSTGIS_RASTER=OFF" "-DGDAL_USE_GEOS=OFF" "-DCMAKE_INSTALL_PREFIX=/builds/gstr/gst/target/test-jenkins/build/gdal-src-5f4a8daa0fa94874/out" "-DCMAKE_C_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_C_COMPILER=/usr/bin/cc" "-DCMAKE_CXX_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_CXX_COMPILER=/usr/bin/c++" "-DCMAKE_ASM_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_ASM_COMPILER=/usr/bin/cc" "-DCMAKE_BUILD_TYPE=Release" -- Could NOT find SWIG (missing: SWIG_EXECUTABLE SWIG_DIR) -- Could NOT find Python (missing: Python_INCLUDE_DIRS Python_LIBRARIES Python_NumPy_INCLUDE_DIRS Development NumPy Development.Module Development.Embed) (found suitable version "3.11.2", minimum required is "3.8") -- checking if sprintf can be overloaded for GDAL compilation -- GDAL_VERSION = 3.9.0dev -- GDAL_ABI_FULL_VERSION = 35.3.9.0 -- GDAL_SOVERSION = 35 -- Could NOT find ODBC (missing: ODBC_LIBRARY ODBC_INCLUDE_DIR ODBCINST) -- Could NOT find ODBCCPP (missing: ODBCCPP_LIBRARY ODBCCPP_INCLUDE_DIR) -- Could NOT find MSSQL_ODBC (missing: MSSQL_ODBC_LIBRARY MSSQL_ODBC_INCLUDE_DIR MSSQL_ODBC_VERSION) -- MYSQL has been found, but is disabled due to GDAL_USE_EXTERNAL_LIBS=OFF. Enable it by setting GDAL_USE_MYSQL=ON -- CURL has been found, but is disabled due to GDAL_USE_EXTERNAL_LIBS=OFF. Enable it by setting GDAL_USE_CURL=ON -- ICONV has been found, but is disabled due to GDAL_USE_EXTERNAL_LIBS=OFF. Enable it by setting GDAL_USE_ICONV=ON -- LIBXML2 has been found, but is disabled due to GDAL_USE_EXTERNAL_LIBS=OFF. Enable it by setting GDAL_USE_LIBXML2=ON -- EXPAT has been found, but is disabled due to GDAL_USE_EXTERNAL_LIBS=OFF. Enable it by setting GDAL_USE_EXPAT=ON -- Failed to find XercesC (missing: XercesC_LIBRARY XercesC_INCLUDE_DIR XercesC_VERSION) -- ZLIB has been found, but is disabled due to GDAL_USE_EXTERNAL_LIBS=OFF. Enable it by setting GDAL_USE_ZLIB=ON -- DEFLATE has been found, but is disabled due to GDAL_USE_EXTERNAL_LIBS=OFF. Enable it by setting GDAL_USE_DEFLATE=ON -- OPENSSL has been found, but is disabled due to GDAL_USE_EXTERNAL_LIBS=OFF. Enable it by setting GDAL_USE_OPENSSL=ON -- Could NOT find CryptoPP (missing: CRYPTOPP_LIBRARY CRYPTOPP_TEST_KNOWNBUG CRYPTOPP_INCLUDE_DIR) -- Could NOT find PROJ (missing: PROJ_DIR) -- TIFF has been found, but is disabled due to GDAL_USE_EXTERNAL_LIBS=OFF. Enable it by setting GDAL_USE_TIFF=ON -- ZSTD has been found, but is disabled due to GDAL_USE_EXTERNAL_LIBS=OFF. Enable it by setting GDAL_USE_ZSTD=ON -- Could NOT find SFCGAL (missing: SFCGAL_LIBRARY SFCGAL_INCLUDE_DIR) -- Could NOT find GeoTIFF (missing: GeoTIFF_DIR) -- Could NOT find GeoTIFF (missing: GEOTIFF_LIBRARY GEOTIFF_INCLUDE_DIR) -- PNG has been found, but is disabled due to GDAL_USE_EXTERNAL_LIBS=OFF. Enable it by setting GDAL_USE_PNG=ON -- JPEG has been found, but is disabled due to GDAL_USE_EXTERNAL_LIBS=OFF. Enable it by setting GDAL_USE_JPEG=ON -- Could NOT find GIF (missing: GIF_LIBRARY GIF_INCLUDE_DIR) -- Could NOT find JSONC (missing: JSONC_DIR) -- Could NOT find JSONC (missing: JSONC_LIBRARY JSONC_INCLUDE_DIR) -- Could NOT find OpenCAD (missing: OPENCAD_LIBRARY OPENCAD_INCLUDE_DIR) -- Could NOT find QHULL (missing: QHULL_LIBRARY QHULL_INCLUDE_DIR) -- LERC has been found, but is disabled due to GDAL_USE_EXTERNAL_LIBS=OFF. Enable it by setting GDAL_USE_LERC=ON -- Could NOT find BRUNSLI (missing: BRUNSLI_ENC_LIB BRUNSLI_DEC_LIB BRUNSLI_INCLUDE_DIR) -- Could NOT find libQB3 (missing: libQB3_DIR) -- Could NOT find Shapelib (missing: Shapelib_INCLUDE_DIR Shapelib_LIBRARY) -- PCRE2 has been found, but is disabled due to GDAL_USE_EXTERNAL_LIBS=OFF. Enable it by setting GDAL_USE_PCRE2=ON -- Could NOT find PCRE (missing: PCRE_LIBRARY PCRE_INCLUDE_DIR) -- Invalidating SQLite3_HAS_COLUMN_METADATA -- Invalidating SQLite3_HAS_LOAD_EXTENSION -- Invalidating SQLite3_HAS_MUTEX_ALLOC -- Invalidating SQLite3_HAS_NON_DEPRECATED_AUTO_EXTENSION -- Invalidating SQLite3_HAS_OPEN -- Invalidating SQLite3_HAS_PROGRESS_HANDLER -- Invalidating SQLite3_HAS_RTREE -- Looking for sqlite3_mutex_alloc -- Looking for sqlite3_mutex_alloc - not found -- Looking for sqlite3_column_table_name -- Looking for sqlite3_column_table_name - not found -- Looking for sqlite3_rtree_query_callback -- Looking for sqlite3_rtree_query_callback - not found -- Looking for sqlite3_load_extension -- Looking for sqlite3_load_extension - not found -- Looking for sqlite3_progress_handler -- Looking for sqlite3_progress_handler - not found -- Performing Test SQLite3_HAS_NON_DEPRECATED_AUTO_EXTENSION -- Performing Test SQLite3_HAS_NON_DEPRECATED_AUTO_EXTENSION - Failed -- Configuring incomplete, errors occurred! See also "/builds/gstr/gst/target/test-jenkins/build/gdal-src-5f4a8daa0fa94874/out/build/CMakeFiles/CMakeOutput.log". See also "/builds/gstr/gst/target/test-jenkins/build/gdal-src-5f4a8daa0fa94874/out/build/CMakeFiles/CMakeError.log". --- stderr CMake Error at cmake/helpers/CheckDependentLibraries.cmake:569 (message): //builds/gstr/gst/target/test-jenkins/build/libsqlite3-sys-2a514ff13e459a59/out/libsqlite3.a lacks mutex support! Access to SQLite3 databases from multiple threads will be unsafe. Define the ACCEPT_MISSING_SQLITE3_MUTEX_ALLOC:BOOL=ON CMake variable if you want to build despite this limitation. Call Stack (most recent call first): gdal.cmake:266 (include) CMakeLists.txt:240 (include) ```

If you search the attached CMakeOutput.log and CMakeError.log output for sqlite3_mutex_alloc you find 4 compile tests. Two of them seems to link libm and pass and two of them doesn't seem to link libm and fail. At least for me the relevant CMake code doesn't look like it should ever hit this check if we got as far as there without knowing which library should be linked in addition to libsqlite3. Therefore I'm a bit unsure what's wrong here.

https://github.com/OSGeo/gdal/blob/5573b18034fc8c6b84c709bef92a4658c7d738af/cmake/modules/packages/FindSQLite3.cmake#L74-L140

Steps to reproduce the issue

It's hard to say what exactly is required to reproduce this issue. It's something we observe in our CI system is there is some load on the system, ie. several (separated) builds running in parallel.

We use a rust crate that builds gdal from source to statically link the library in the end: https://github.com/GiGainfosystems/gdal/commit/593a6b59bd183dfc74ffff355593088462d292d8

Versions and provenance

This happens with https://github.com/OSGeo/gdal/tree/bb5ea618e23e498e79e8b8e6bd7215ad94bb2103 (3.9.0)

Additional context

No response

rouault commented 1 month ago

The -- Invalidating SQLite3_HAS_COLUMN_METADATA -- Invalidating SQLite3_HAS_LOAD_EXTENSION -- Invalidating SQLite3_HAS_MUTEX_ALLOC -- Invalidating SQLite3_HAS_NON_DEPRECATED_AUTO_EXTENSION -- Invalidating SQLite3_HAS_OPEN -- Invalidating SQLite3_HAS_PROGRESS_HANDLER -- Invalidating SQLite3_HAS_RTREE messages in the log seem to be an indication that you are not building into a fresh build directory, which is not recommended , especially if the outside environment might have changed between builds

weiznich commented 1 month ago

Thanks for the fast response. The outside environment does not change between builds it's a static docker image that get restarted for every build. As already written: It is using a libsqlite3 version that is build exactly with the same flags every time, so it certainly doesn't change any dependency of libsqlite3 in between those builds.

rouault commented 1 month ago

The outside environment does not change between builds it's a static docker image that get restarted for every build.

but it has already a previous GDAL build directory ? otherwise I can't explain the Invalidating messages. Could you try to do fresh builds to see if that makes a difference ?

weiznich commented 1 month ago

It seems that cleaning the build directory works around this issue. I do consider that only as a workaround as that has an obvious negative impact on build times.