search

mgba-emu / mgba

mGBA Game Boy Advance Emulator
https://mgba.io/
Mozilla Public License 2.0
5.73k stars 794 forks source link

Build issue on (cross-compile) MinGW related to CFFI (pycparser) for Python bindings #2360

Closed nb-programmer closed 1 month ago

nb-programmer commented 2 years ago

Scratching my head over this for over a week now, seems to be related to CFFI trying to parse MinGW headers. In particular, I'm trying to build Python bindings for use in a different project. I tried building on MSYS2 with mingw64 environment (after installing all packages), but gave the same results as the below attempt on an Arch Linux machine with MinGW-w64 cross compiler.

Here's what I've done till now: CMake command (in directory buildwin inside the source code):

$ cmake -DCMAKE_TOOLCHAIN_FILE=/usr/share/mingw/toolchain-x86_64-w64-mingw32.cmake -DBUILD_PYTHON=ON -DUSE_FFMPEG=OFF -DUSE_DEBUGGERS=OFF -DUSE_DISCORD_RPC=OFF -DUSE_GDB_STUB=OFF -DENABLE_SCRIPTING=OFF -DBUILD_QT=OFF
 -DBUILD_SDL=OFF -DBUILD_LIBRETRO=OFF -DUSE_SQLITE3=OFF -DUSE_ELF=OFF -DUSE_ZLIB=OFF -DUSE_LZMA=OFF -DBUILD_GLES2=OFF -D
BUILD_GLES3=OFF -DBUILD_GL=OFF -DDISABLE_FRONTENDS=ON ..

-- The C compiler identification is GNU 11.2.0
-- The CXX compiler identification is GNU 11.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/x86_64-w64-mingw32-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/x86_64-w64-mingw32-g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PkgConfig: /usr/bin/pkg-config (found version "1.8.0")
-- Looking for strdup
-- Looking for strdup - not found
-- Looking for strndup
-- Looking for strndup - not found
-- Looking for strlcpy
-- Looking for strlcpy - not found
-- Looking for vasprintf
-- Looking for vasprintf - not found
-- Looking for localtime_r
-- Looking for localtime_r - not found
-- Looking for include file xlocale.h
-- Looking for include file xlocale.h - not found
-- Looking for snprintf_l
-- Looking for snprintf_l - not found
-- Looking for strtof_l
-- Looking for strtof_l - not found
-- Looking for newlocale
-- Looking for newlocale - not found
-- Looking for freelocale
-- Looking for freelocale - not found
-- Looking for uselocale
-- Looking for uselocale - not found
-- Looking for setlocale
-- Looking for setlocale - not found
-- Looking for chmod
-- Looking for chmod - not found
-- Looking for umask
-- Looking for umask - not found
-- Checking for one of the modules 'minizip'
CMake Warning at src/platform/cmake/FindFeature.cmake:51 (message):
  Requested module minizip missing for feature USE_MINIZIP.  Feature
  disabled.
Call Stack (most recent call first):
  CMakeLists.txt:518 (find_feature)

-- Checking for one of the modules 'PNG'
CMake Warning at src/platform/cmake/FindFeature.cmake:51 (message):
  Requested module PNG missing for feature USE_PNG.  Feature disabled.
Call Stack (most recent call first):
  CMakeLists.txt:519 (find_feature)

-- Checking for one of the modules 'libzip'
CMake Warning at src/platform/cmake/FindFeature.cmake:51 (message):
  Requested module libzip missing for feature USE_LIBZIP.  Feature disabled.
Call Stack (most recent call first):
  CMakeLists.txt:520 (find_feature)

-- Looking for crc32
-- Looking for crc32 - not found
-- Found PythonInterp: /usr/bin/python (found version "3.9.7")
-- Could NOT find PythonLibs (missing: PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS)
-- Build type: Release
-- Platforms:
--      Game Boy Advance: ON
--      Game Boy: ON
-- Features:
--      Debuggers: OFF
--      GDB stub: OFF
--      GIF/Video recording: OFF
--      Screenshot/advanced savestate support: OFF
--      ZIP support: OFF
--      7-Zip support: OFF
--      SQLite3 game database: OFF
--      ELF loading support: OFF
--      Discord Rich Presence support: OFF
--      OpenGL support: libepoxy
-- Frontends:
--      Qt: OFF
--      SDL (): OFF
--      Profiling: OFF
--      Test harness: OFF
--      Test suite: OFF
--      Video test suite: OFF
--      Python bindings: ON
--      Examples: OFF
-- Cores:
--      Libretro core: OFF
-- Libraries:
--      Static: OFF
--      Shared: ON
-- Configuring done
-- Generating done
-- Build files have been written to: /mnt/564E192F4E1908FF/Downloads/mgba/mgba-0.9.2/buildwin

Note: The huge number of flags is because I was trying to reduce as many unnecessary modules as possible. It produces the same issue even with the other flags removed.

Then I ran make in "python" directory:

$ cd python/
$ make
[  0%] Built target mgba-version-info
<snip>
[ 93%] Linking C shared library libmgba.dll
[ 93%] Built target mgba
[ 94%] Generating lib.c
In file included from /usr/x86_64-w64-mingw32/include/corecrt.h:10,
                 from /usr/x86_64-w64-mingw32/include/crtdefs.h:10,
                 from /usr/x86_64-w64-mingw32/include/limits.h:6,
                 from /mnt/564E192F4E1908FF/Downloads/mgba/mgba-0.9.2/src/platform/python/_builder.h:35:
/usr/x86_64-w64-mingw32/include/_mingw.h:272:2: error: #error Only Win32 target is supported!
  272 | #error Only Win32 target is supported!
      |  ^~~~~
Traceback (most recent call last):
  File "/mnt/564E192F4E1908FF/Downloads/mgba/mgba-0.9.2/src/platform/python/_builder.py", line 59, in <module>
    preprocessed = subprocess.check_output(cpp + ["-fno-inline", "-P"] + cppflags + [os.path.join(pydir, "_builder.h")], universal_newlines=True)
  File "/usr/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cc', '-E', '-fno-inline', '-P', '-I/mnt/564E192F4E1908FF/Downloads/mgba/mgba-0.9.2/buildwin/include', '-I/mnt/564E192F4E1908FF/Downloads/mgba/mgba-0.9.2/src', '-I/mnt/564E192F4E1908FF/Downloads/mgba/mgba-0.9.2/include', '-I/usr/x86_64-w64-mingw32/include', '-I/mnt/564E192F4E1908FF/Downloads/mgba/mgba-0.9.2/src/platform/python/../../../include', '-I/mnt/564E192F4E1908FF/Downloads/mgba/mgba-0.9.2/src/platform/python/../..', '-I/mnt/564E192F4E1908FF/Downloads/mgba/mgba-0.9.2/buildwin/python/..', '/mnt/564E192F4E1908FF/Downloads/mgba/mgba-0.9.2/src/platform/python/_builder.h']' returned non-zero exit status 1.
make[2]: *** [python/CMakeFiles/mgba-pylib.dir/build.make:83: python/lib.c] Error 1
make[1]: *** [CMakeFiles/Makefile2:347: python/CMakeFiles/mgba-pylib.dir/all] Error 2
make: *** [Makefile:166: all] Error 2

It failed because the _builder.py uses "cc -E" by default, which is native GCC's preprocessor, not MinGW's preprocessor.

Try again with correct MinGW C preprocessor with CPP variable:

$ CPP=x86_64-w64-mingw32-cpp make
[  0%] Built target mgba-version-info
[ 93%] Built target mgba
[ 94%] Generating lib.c
/usr/lib/python3.9/site-packages/cffi/cparser.py:154: UserWarning: String literal found in cdef() or type source. String literals are ignored here, but you should remove them anyway because some character sequences confuse pre-parsing.
  warnings.warn("String literal found in cdef() or type source. "
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/cffi/cparser.py", line 336, in _parse
    ast = _get_parser().parse(fullcsource)
  File "/usr/lib/python3.9/site-packages/pycparser/c_parser.py", line 149, in parse
    return self.cparser.parse(
  File "/usr/lib/python3.9/site-packages/pycparser/ply/yacc.py", line 331, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/usr/lib/python3.9/site-packages/pycparser/ply/yacc.py", line 1199, in parseopt_notrack
    tok = call_errorfunc(self.errorfunc, errtoken, self)
  File "/usr/lib/python3.9/site-packages/pycparser/ply/yacc.py", line 193, in call_errorfunc
    r = errorfunc(token)
  File "/usr/lib/python3.9/site-packages/pycparser/c_parser.py", line 1858, in p_error
    self._parse_error(
  File "/usr/lib/python3.9/site-packages/pycparser/plyparser.py", line 67, in _parse_error
    raise ParseError("%s: %s" % (coord, msg))
pycparser.plyparser.ParseError: <cdef source string>:10:27: before: __gnuc_va_list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/564E192F4E1908FF/Downloads/mgba/mgba-0.9.2/src/platform/python/_builder.py", line 67, in <module>
    ffi.cdef('\n'.join(lines))
  File "/usr/lib/python3.9/site-packages/cffi/api.py", line 112, in cdef
    self._cdef(csource, override=override, packed=packed, pack=pack)
  File "/usr/lib/python3.9/site-packages/cffi/api.py", line 126, in _cdef
    self._parser.parse(csource, override=override, **options)
  File "/usr/lib/python3.9/site-packages/cffi/cparser.py", line 389, in parse
    self._internal_parse(csource)
  File "/usr/lib/python3.9/site-packages/cffi/cparser.py", line 394, in _internal_parse
    ast, macros, csource = self._parse(csource)
  File "/usr/lib/python3.9/site-packages/cffi/cparser.py", line 338, in _parse
    self.convert_pycparser_error(e, csource)
  File "/usr/lib/python3.9/site-packages/cffi/cparser.py", line 367, in convert_pycparser_error
    raise CDefError(msg)
cffi.CDefError: cannot parse "typedef __builtin_va_list __gnuc_va_list;"
<cdef source string>:10:27: before: __gnuc_va_list
make[2]: *** [python/CMakeFiles/mgba-pylib.dir/build.make:83: python/lib.c] Error 1
make[1]: *** [CMakeFiles/Makefile2:347: python/CMakeFiles/mgba-pylib.dir/all] Error 2
make: *** [Makefile:166: all] Error 2

Here is where I got stuck. CFFI's pycparser module fails to determine GCC's built-in symbol "__builtin_va_list", and more (if I manually define it, it fails again with more symbols). This is because the _builder.h file loads , which, in MinGW loads "_mingw.h" which has loads of MinGW related typedefs using compiler built-in symbols. I tried to use pycparser's fake libc headers, but still more things remain to be patched to compile completely

Environment details:

Let me know if any other detail is needed

nb-programmer commented 2 years ago

https://stackoverflow.com/a/56170034/12887350 This is the issue when using MSYS2 and CMAKE

RetroEdit commented 1 month ago

Closing, as the Python API is now deprecated in favor of the newer Lua API. Python is planned as an expansion to the new scripting system eventually (#3048).

Sadly, I don't see this being resolved here: the GitHub issue tracker is more intended for bugs/feature requests and often not well-suited to support-type issues; the Discord server can often work better.