search

OSGeo / gdal

GDAL is an open source MIT licensed translator library for raster and vector geospatial data formats.
https://gdal.org
Other
4.68k stars 2.47k forks source link

Unable to Correctly Read mdb Files on M1 MacOS #9816

Open PISERERER opened 3 months ago

PISERERER commented 3 months ago

What is the bug?

I compiled GDAL 3.8.5 with support for the PGeo driver on an M1 MacOS, but when I ran the command ogrinfo --debug on -if PGeo 640122HLX_2006Y00_SV050_P.mdb, I received the following debug information:

ODBC: Declaration of Microsoft Access Driver found in /etc/odbcinst.ini
ODBC: EstablishSession(DRIVER=Microsoft Access Driver (*.mdb, *.accdb);DBQ=640122HLX_2006Y00_SV050_P.mdb)
ODBC: SQLDriverConnect(DRIVER=Microsoft Access Driver (*.mdb, *.accdb);DBQ=640122HLX_2006Y00_SV050_P.mdb)
ODBC: ... failed:
ODBC: SQLDisconnect()
ODBC: EstablishSession(DRIVER=Microsoft Access Driver (*.mdb, *.accdb);DBQ="640122HLX_2006Y00_SV050_P.mdb")
ODBC: SQLDriverConnect(DRIVER=Microsoft Access Driver (*.mdb, *.accdb);DBQ="640122HLX_2006Y00_SV050_P.mdb")
ODBC: ... failed:
ODBC: SQLDisconnect()
ODBC: EstablishSession(DRIVER=Microsoft Access Driver (*.mdb);DBQ=640122HLX_2006Y00_SV050_P.mdb)
ODBC: SQLDriverConnect(DRIVER=Microsoft Access Driver (*.mdb);DBQ=640122HLX_2006Y00_SV050_P.mdb)
PGEO: SELECT on GDB_GeomColumns fails, perhaps not a personal geodatabase?

ODBC: SQLDisconnect()

Additionally, the program becomes unresponsive and cannot be stopped, while consuming 100% CPU on a single core (possibly due to an infinite loop?). One particularly valuable piece of information was: 'PGEO: SELECT on GDB_GeomColumns fails, perhaps not a personal geodatabase?' I attempted to trace the GDAL source code and by adding debug information, it seems that the issue occurs when executing the following SQL statement:

    // /ogr/ogrsf_frmts/pgeo/ogrpgeodatasource.cpp

    std::vector<char **> apapszGeomColumns;
    CPLODBCStatement oStmt(&oSession);

    oStmt.Append(
        "SELECT TableName, FieldName, ShapeType, ExtentLeft, ExtentRight, "
        "ExtentBottom, ExtentTop, SRID, HasZ, HasM FROM GDB_GeomColumns");

    if (!oStmt.ExecuteSQL())
    {
        CPLDebug("PGEO",
                 "SELECT on GDB_GeomColumns fails, perhaps not a personal "
                 "geodatabase?\n%s",
                 oSession.GetLastError());
        return FALSE;
    }

Continuing the trace, it appears that the SQLExecDirect method returns -2, but no valuable error information is captured. As I am not very familiar with C++, I am unable to further trace the issue. It is clear, however, that the SQLExecDirect method is from unixODBC (if I understand correctly).

    //  /port/cpl_odbc.cpp

    // SQL_NTS=-3 is a valid value for SQLExecDirect.
    // coverity[negative_returns]
    if (Failed(SQLExecDirect(
            m_hStmt, reinterpret_cast<SQLCHAR *>(m_pszStatement), SQL_NTS)))
        return FALSE;

    return CollectResultsInfo();

Furthermore, I tried executing the problematic SQL statement directly through unixODBC and encountered no issues. I configured the mdb file in odbc.ini and used the isql command to read it as follows:

➜  testmdb isql -v MyTestAccessDB
+---------------------------------------+
| Connected!                            |
|                                       |
| sql-statement                         |
| help [tablename]                      |
| echo [string]                         |
| quit                                  |
|                                       |
+---------------------------------------+
SQL> SELECT TableName, FieldName, ShapeType, ExtentLeft, ExtentRight, ExtentBottom, ExtentTop, SRID, HasZ, HasM FROM GDB_GeomColumns
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+-----------+------------+-------------+-----------+------------+-----+-----+
| TableName                                                                                                                                                                                                                                                                                                   | FieldName                                                                                                                                                                                                                                                                                                   | ShapeType  | ExtentLeft| ExtentRight| ExtentBottom| ExtentTop | SRID       | HasZ| HasM|
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+-----------+------------+-------------+-----------+------------+-----+-----+
| GDB_Items                                                                                                                                                                                                                                                                                                   | Shape                                                                                                                                                                                                                                                                                                       | 4          | 106.062013| 106.121651 | 38.832024999| 38.855929 | 1          | 0   | 0   |
| MPID                                                                                                                                                                                                                                                                                                        | Shape                                                                                                                                                                                                                                                                                                       | 4          | 106.062013| 106.1216514| 38.832024971| 38.8559290| 2          | 0   | 0   |
| MBII                                                                                                                                                                                                                                                                                                        | Shape                                                                                                                                                                                                                                                                                                       | 4          | 106.062013| 106.1216514| 38.832024971| 38.8559290| 2          | 0   | 0   |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+-----------+------------+-------------+-----------+------------+-----+-----+
SQLRowCount returns 3
3 rows fetched

I am confident that there is no issue with the mdb file itself, as I not only compiled GDAL on my machine but also added the PGeo driver to the official ARM version of the GDAL Docker image. I then executed the same command against the same mdb file within the container and obtained the correct results. The output was as follows:

ODBC: Declaration of Microsoft Access Driver found in /etc/odbcinst.ini
ODBC: EstablishSession(DRIVER=Microsoft Access Driver (*.mdb, *.accdb);DBQ=640122HLX_2006Y00_SV050_P.mdb)
ODBC: SQLDriverConnect(DRIVER=Microsoft Access Driver (*.mdb, *.accdb);DBQ=640122HLX_2006Y00_SV050_P.mdb)
ODBC: ... failed:
ODBC: SQLDisconnect()
ODBC: EstablishSession(DRIVER=Microsoft Access Driver (*.mdb, *.accdb);DBQ="640122HLX_2006Y00_SV050_P.mdb")
ODBC: SQLDriverConnect(DRIVER=Microsoft Access Driver (*.mdb, *.accdb);DBQ="640122HLX_2006Y00_SV050_P.mdb")
ODBC: ... failed:
ODBC: SQLDisconnect()
ODBC: EstablishSession(DRIVER=Microsoft Access Driver (*.mdb);DBQ=640122HLX_2006Y00_SV050_P.mdb)
ODBC: SQLDriverConnect(DRIVER=Microsoft Access Driver (*.mdb);DBQ=640122HLX_2006Y00_SV050_P.mdb)
ODBC: CatalogNameL: (null)
Schema name: (null)
PGeo: MPID: no primary key
PGeo: MBII: no primary key
ODBC: SQLDisconnect()
ODBC: EstablishSession(DRIVER=Microsoft Access Driver (*.mdb, *.accdb);DBQ=640122HLX_2006Y00_SV050_P.mdb)
ODBC: SQLDriverConnect(DRIVER=Microsoft Access Driver (*.mdb, *.accdb);DBQ=640122HLX_2006Y00_SV050_P.mdb)
ODBC: ... failed:
ODBC: SQLDisconnect()
ODBC: EstablishSession(DRIVER=Microsoft Access Driver (*.mdb, *.accdb);DBQ="640122HLX_2006Y00_SV050_P.mdb")
ODBC: SQLDriverConnect(DRIVER=Microsoft Access Driver (*.mdb, *.accdb);DBQ="640122HLX_2006Y00_SV050_P.mdb")
ODBC: ... failed:
ODBC: SQLDisconnect()
ODBC: EstablishSession(DRIVER=Microsoft Access Driver (*.mdb);DBQ=640122HLX_2006Y00_SV050_P.mdb)
ODBC: SQLDriverConnect(DRIVER=Microsoft Access Driver (*.mdb);DBQ=640122HLX_2006Y00_SV050_P.mdb)
ODBC: CatalogNameL: (null)
Schema name: (null)
PGeo: MPID: no primary key
PGeo: MBII: no primary key
GDAL: GDALOpen(640122HLX_2006Y00_SV050_P.mdb, this=0xaaaae6417eb0) succeeds as PGeo.
INFO: Open of `640122HLX_2006Y00_SV050_P.mdb'
      using driver `PGeo' successful.
OGR: GetLayerCount() = 2

1: MPID (Multi Polygon)
2: MBII (Multi Polygon)
ODBC: SQLDisconnect()
GDAL: GDALClose(640122HLX_2006Y00_SV050_P.mdb, this=0xaaaae6417eb0)

Steps to reproduce the issue

Install unixODBC, mdbtools, etc., compile GDAL 3.8.5, and execute the command: ogrinfo --debug on -if PGeo xxx.mdb. The compilation command is as follows:

cd gdal-3.8.5

mkdir build
cd build

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/Users/yuziqun/main/software/gdal-3.8.5-nokml -DCMAKE_INSTALL_RPATH=/Users/yuziqun/main/software/gdal-3.8.5-nokml/lib -DBUILD_PYTHON_BINDINGS=OFF -DGDAL_USE_LIBKML=OFF -DCMAKE_DISABLE_FIND_PACKAGE_Arrow=ON ..

cmake --build . -- -j6

cmake --build . --target install

Versions and provenance

M1 pro MacOS 14.4.1

➜  build ogrinfo --version
GDAL 3.8.5, released 2024/04/02
➜  build isql --version
unixODBC 2.3.12
➜  build mdb-ver --version
mdbtools v1.0.0

Additional context

No response

rouault commented 3 months ago

as I not only compiled GDAL on my machine but also added the PGeo driver to the official ARM version of the GDAL Docker image

which docker image exactly? We have 4 variants. So this does work on Arm with a GDAL Docker image, but not in your native environment? Personally, I won't be able to investigate now owing a Mac

PISERERER commented 3 months ago

as I not only compiled GDAL on my machine but also added the PGeo driver to the official ARM version of the GDAL Docker image

which docker image exactly? We have 4 variants. So this does work on Arm with a GDAL Docker image, but not in your native environment? Personally, I won't be able to investigate now owing a Mac

Yes, it runs normally in the ARM-based Docker image container, which is also running on Docker on my Mac, but there are issues in my local MacOS ARM environment. I am using this image: https://github.com/OSGeo/gdal/pkgs/container/gdal/180417381?tag=ubuntu-full-3.8.4 I noticed one difference: in the image, the unixODBC version is 2.3.9.