search

Exiv2 / exiv2

Image metadata library and tools
http://www.exiv2.org/
Other
911 stars 280 forks source link

Update README.md to explain cross-compiling #922

Closed clanmills closed 4 years ago

clanmills commented 5 years ago

When I wrote README.md for v0.27, I had never succeeded in cross-compiling for MinGW from Linux. I believe this is now working and Dan has a CI job.

I hope to refresh my brain about this and get it documented for v0.27.2. For sure for v0.27.3

clanmills commented 5 years ago

@piponazo I'll deal with README.md and README-CONAN.md. I believe most of the new documentation will go into README-CONAN.md and will be referenced in README.md

I have not been able to get CMake to find and use the expat and zlib which were cross-compiled by conan, so I'm going to defer this for v0.27.3. Here's what I've done on Ubuntu 18.04

Install pip3 and conan:

1) Install pip3 $ sudo apt install python3-pip 2) Install conan $ sudo pip3 install conan

Configure for cross-platform development

1) Install the cross-platform tools $ sudo apt-get install g++-mingw-w64 gcc-mingw-w64 https://docs.conan.io/en/latest/systems_cross_building/cross_building.html 2) Add this file to \<exiv2dir>/cmake/mingw-w64-x86_64.cmake

# Sample toolchain file for building for Windows from an Ubuntu Linux system.
#
# Typical usage:
#    *) install cross compiler: `sudo apt-get install mingw-w64`
#    *) cd build
#    *) cmake -DCMAKE_TOOLCHAIN_FILE=~/mingw-w64-x86_64.cmake ..

set(CMAKE_SYSTEM_NAME Windows)
set(TOOLCHAIN_PREFIX x86_64-w64-mingw32)

# cross compilers to use for C, C++ and Fortran
set(CMAKE_C_COMPILER ${TOOLCHAIN_PREFIX}-gcc)
set(CMAKE_CXX_COMPILER ${TOOLCHAIN_PREFIX}-g++)
set(CMAKE_Fortran_COMPILER ${TOOLCHAIN_PREFIX}-gfortran)
set(CMAKE_RC_COMPILER ${TOOLCHAIN_PREFIX}-windres)

# target environment on the build host system
set(CMAKE_FIND_ROOT_PATH /usr/${TOOLCHAIN_PREFIX})

# modify default behavior of FIND_XXX() commands
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

3) Add a new profile to ~/.conan/profiles/linux-to-win64

toolchain=/usr/x86_64-w64-mingw32 # Adjust this path
target_host=x86_64-w64-mingw32
cc_compiler=gcc
cxx_compiler=g++

[env]
CONAN_CMAKE_FIND_ROOT_PATH=$toolchain
CHOST=$target_host
AR=$target_host-ar
AS=$target_host-as
RANLIB=$target_host-ranlib
CC=$target_host-$cc_compiler
CXX=$target_host-$cxx_compiler
STRIP=$target_host-strip
RC=$target_host-windres

[settings]
# We are building in Ubuntu Linux
os_build=Linux
arch_build=x86_64

# We are cross-building to Windows
os=Windows
arch=x86_64
compiler=gcc

# Adjust to the gcc version of your MinGW package
compiler.version=7.3
compiler.libcxx=libstdc++11
build_type=Release

Build dependencies, run cmake, and build:

1) $ cd <exiv2dir> ; mkdir MinGW_build ; cd MinGW_build 2) $ conan install .. --profile linux-to-win64 --build missing 3) $ cmake .. -DCMAKE_TOOLCHAIN_FILE=../cmake/mingw-w64-x86_64.cmake

rmills@rmillsmm-ubuntu:~/gnu/github/exiv2/0.27-maintenance/MinGW-build$ cmake .. -DCMAKE_TOOLCHAIN_FILE=../cmake/mingw-w64-x86_64.cmake 
-- Conan: Using cmake targets configuration
-- Library zlib found /home/rmills/.conan/data/zlib/1.2.11/conan/stable/package/040db29ed48cb6d2bb9f4c6d13e74d5e319c7ad8/lib/libzlib.a
-- Library gmock_main found /home/rmills/.conan/data/gtest/1.8.1/bincrafters/stable/package/dc6599c77833e79bb142050659ef8269e4e1d8c5/lib/libgmock_main.dll.a
-- Library gmock found /home/rmills/.conan/data/gtest/1.8.1/bincrafters/stable/package/dc6599c77833e79bb142050659ef8269e4e1d8c5/lib/libgmock.dll.a
-- Library gtest found /home/rmills/.conan/data/gtest/1.8.1/bincrafters/stable/package/dc6599c77833e79bb142050659ef8269e4e1d8c5/lib/libgtest.dll.a
-- Library expat found /home/rmills/.conan/data/Expat/2.2.6/pix4d/stable/package/776dc1175d2af830f77a93cb1550e2a530879c82/lib/libexpat.a
-- Conan: Adjusting language standard
-- Current conanbuildinfo.cmake directory: /home/rmills/gnu/github/exiv2/0.27-maintenance/MinGW-build
-- Conan: Compiler GCC>=5, checking major version 7.3
-- Conan: Checking correct version: 7.3
CMake Error at /usr/share/cmake-3.10/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find EXPAT (missing: EXPAT_LIBRARY EXPAT_INCLUDE_DIR)
Call Stack (most recent call first):
  /usr/share/cmake-3.10/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.10/Modules/FindEXPAT.cmake:61 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  cmake/findDependencies.cmake:31 (find_package)
  CMakeLists.txt:61 (include)

-- Configuring incomplete, errors occurred!
See also "/home/rmills/gnu/github/exiv2/0.27-maintenance/MinGW-build/CMakeFiles/CMakeOutput.log".
See also "/home/rmills/gnu/github/exiv2/0.27-maintenance/MinGW-build/CMakeFiles/CMakeError.log".
rmills@rmillsmm-ubuntu:~/gnu/github/exiv2/0.27-maintenance/MinGW-build$

3a) $ cmake .. -DCMAKE_TOOLCHAIN_FILE=../cmake/mingw-w64-x86_64.cmake -DEXIV2_ENABLE_XMP=Off -DEXIV2_ENABLE_PNG=Off 4) $ make It's built:

rmills@rmillsmm-ubuntu:~/gnu/github/exiv2/0.27-maintenance/MinGW-build$ ls -l bin/exiv2.exe
-rwxrwxr-x 1 rmills rmills 1185810 Jun 23 20:54 bin/exiv2.exe
rmills@rmillsmm-ubuntu:~/gnu/github/exiv2/0.27-maintenance/MinGW-build$

Testing the build

Don't know yet. To be investigated. It can be tested in MinGW in much the same way as I test Visual Studio builds.

piponazo commented 5 years ago

Sorry, I did not see this until now. I remember that I had also some issues with some of the dependencies when I tried to cross-compile from Ubuntu 18.04 to Windows with MinGW, but I think at the end I could solve them. I'll take a look to it at some point.

D4N commented 5 years ago

Imho the simplest way how to cross compile from Linux with MinGW is to use Fedora (either grab the vagrant box or a container via podman pull docker.io/library/fedora) as we do on GitLab:

$ sudo dnf install mingw64-gcc-c++ mingw64-filesystem mingw64-expat mingw64-zlib cmake make
$ mkdir build && pushd build
$ mingw64-cmake -DEXIV2_TEAM_EXTRA_WARNINGS=ON -DEXIV2_ENABLE_VIDEO=ON -DEXIV2_ENABLE_WEBREADY=ON -DEXIV2_ENABLE_WIN_UNICODE=ON -DBUILD_SHARED_LIBS=OFF ..
$ make -j $(nproc)
$ popd

That's literally it. The only caveat is that shared libraries don't work due to the issues with the Error type, which I haven't solved yet in #685 and #779.

clanmills commented 5 years ago

I can't get this to work on Ubuntu for the following reasons: 1) Conan doesn't install the expat dependency. 2) The file cmake/mingw-w64-x86_64.cmake isn't in the repos. 3) Dan's recipe requires mingw-cmake which does not exist in apt. 4) Building "static only" is of limited usefulness. 5) I'd like to document how to test this on Windows or using wine on Linux.

I'm going to defer this for v0.27.3.

D4N commented 5 years ago

Robin Mills notifications@github.com writes:

I can't get this to work on Ubuntu for the following reasons: 1) Conan doesn't install the expat dependency. 2) The file cmake/mingw-w64-x86_64.cmake isn't in the repos. 3) Dan's recipe requires mingw-cmake which does not exist in apt.

Because that is only available on Fedora. If you want a painless solution: use Fedora, grab the same packages as on GitLab and it should just work (no need to mess with Ubuntu, conan and cmake toolchain files).

4) Building "static only" is of limited usefulness.

This should be fixed once #685 makes it into master, no chance for 0.27 though.

5) I'd like to document how to test this on Windows or using wine on Linux.

I'm going to defer this for v0.27.3.

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/Exiv2/exiv2/issues/922#issuecomment-513258177

clanmills commented 5 years ago

Thanks, Dan. That's a bummer.

I would like to get cross-compiling documented for "the dots". I had it working in January with a little magic in contrib and some DLLs which I shouldn't have put into the repos.

I'll revisit this and look at Fedora for 0.27.3. Or maybe I'll focus on the book. We'll see.

clanmills commented 4 years ago

I've had another visit to this topic and installed Fedora 31.1.9 on a fresh VM.

Dan's instructions here are correct:https://github.com/Exiv2/exiv2/issues/922#issuecomment-508567229

I had some work to perform to install zlib1-devel and expat-devel on x86_64 to build native, however it built smoothly and passed the test framework. Then I used Dan's instructions to cross-compile, and indeed bin/exiv2.exe was generated.

I haven't figured out how to configure the Fedora Network to "see" rmillsmm-w10 (a Windows VM), so I zipped the 0.27-maintenance directory, and copied it to c:\msys64\home\rmills\temp\0.27-maintence. When I attempted to run the test suite ($ make tests), CMake started regenerating the Makefiles to compile with the GCC/msys2 compiler

So, I tried to run the test manually (from information in README.md)

$ cd ~/temp/0.277-maintenance/tests
$ export EXIV2_BINDIR=$PWD/../build_mingw/bin
$ export EXIV2_EXT=.exe
$ make newtests
514 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/test $ make newtests
( cd ../tests ; python3 runner.py --verbose )
test_run (bugfixes.github.test_CVE_2017_1000126.TestCvePoC) ... ERROR
....... everything fails .......

Amazing. It almost worked. So, I tried execute exiv2.exe from the command-prompt:

518 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/test $ ls -l ../build_mingw/bin/exiv2.exe
-rwxr-xr-x 1 rmills rmills 6748907 Apr  6 17:26 ../build_mingw/bin/exiv2.exe
519 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/test $ ../build_mingw/bin/exiv2.exe
C:/msys64/home/rmills/temp/0.27-maintenance/build_mingw/bin/exiv2.exe: error while loading shared libraries: zlib1.dll: cannot open shared object file: No such file or directory
520 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/test $

It's cross compiled to use dynamic dependencies which are not on PATH or LD_LIBRARY_PATH on rmillsmm-w10.

Good Progress. Tomorrow is another day.

clanmills commented 4 years ago

You need to copy 5 DLLs from Fedora as follows:

$ scp /usr/x86_64-w64-mingw32/sys-root/mingw/bin/zlib1.dll rmillsmm-w10:/c:\msys64\home\rmills\temp\0.27-maintenance\build_mingw\bin
$ scp /usr/x86_64-w64-mingw32/sys-root/mingw/bin/libexpat-1.dll  rmillsmm-w10:/c:\msys64\home\rmills\temp\0.27-maintenance\build_mingw\bin
$ scp /usr/x86_64-w64-mingw32/sys-root/mingw/bin/libstdc++-6.dll   rmillsmm-w10:/c:\msys64\home\rmills\temp\0.27-maintenance\build_mingw\bin
$ scp /usr/x86_64-w64-mingw32/sys-root/mingw/bin/libwinpthread-1.dll    rmillsmm-w10:/c:\msys64\home\rmills\temp\0.27-maintenance\build_mingw\bin
$ scp /usr/x86_64-w64-mingw32/sys-root/mingw/bin/libgcc_s_seh-1.dll  rmillsmm-w10:/c:\msys64\home\rmills\temp\0.27-maintenance\build_mingw\bin

The bash tests execute correctly, with the exception of a time issue in conversions.sh which I will not investigate. 

532 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/build_mingw/bin $ export EXIV2_BINDIR=$PWD
533 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/build_mingw/bin $ export EXIV2_EXT=.exe
534 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/build_mingw/bin $ cd ../../test
535 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/test $ $EXIV2_BINDIR/exiv2 -vVg exiv2 -g compiler -g time
exiv2 0.27.3.9
exiv2=0.27.3
compiler=G++
time=17:26:17
package_name=exiv2
executable=C:\msys64\home\rmills\temp\0.27-maintenance\build_mingw\bin\exiv2.exe
have_gmtime_r=0
config_path=C:\Users\rmills\exiv2.ini
xmlns=mediapro:http://ns.iview-multimedia.com/mediapro/1.0/
538 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/test $ make test
mkdir -p tmp
rm -rf tmp/test-failed
Running addmoddel.sh ...
all testcases passed.
Running conversions.sh ...
Files /home/rmills/temp/0.27-maintenance/test/tmp/conversions.out and /home/rmills/temp/0.27-maintenance/test/data/conversions.out differ
165c165
< Exif.Image.DateTime                          Ascii      20  2015:04:17 19:10:22
---
> Exif.Image.DateTime                          Ascii      20  2015:04:18 02:10:22
***
*** conversions.sh result = 3
***
Running exifdata-test.sh ...
all testcases passed.
...

The new tests $ make newtests almost all fail due to different line-endings in the Windows. The reference output in the test suite has LF endings, and the output on Windows has CR-LF.

I've tried to change all the python files without success:

551 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/test $ find ../tests -name "*.py" | xargs unix2dos 
nix2dos: converting file ../tests/bugfixes/redmine/test_issue_922.py to DOS format...
unix2dos: converting file ../tests/bugfixes/redmine/test_issue_935.py to DOS format...
unix2dos: converting file ../tests/bugfixes/redmine/test_issue_937.py to DOS format...
...

That didn't fix it. I'll open a new issue about this and ask @D4N if it's possible to modify the test suite to forgive line-ending mismatches. #1146

D4N commented 4 years ago

Robin Mills notifications@github.com writes:

Running addmoddel.sh ... all testcases passed. Running conversions.sh ... Files /home/rmills/temp/0.27-maintenance/test/tmp/conversions.out and /home/rmills/temp/0.27-maintenance/test/data/conversions.out differ 165c165 < Exif.Image.DateTime Ascii 20 2015:04:17 19:10:22

Exif.Image.DateTime Ascii 20 2015:04:18 02:10:22

Looks like a timezone issue to me.

clanmills commented 4 years ago

@D4N It is a time-zone issue and I've see it before. The test is set up for KL, Malaysia which is GMT+8 and I'm running it in BST (GMT+1). And the difference in output is 7 hours. I'm not going to bother investigating this further.

The task here is to document cross compiling and I have sufficient information to update README.md which will reference this discussion.

If you feel it's important to investigate this TZ issue, please open a new issue.

clanmills commented 4 years ago

The TZ issue has been investigated and resolved with a "tweak" in the test-harness. #485

The "tweak" can be extended to support mingw64:

504 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/build_mingw/bin $ ./exiv2 -vVg platform
exiv2 0.27.3.9
platform=mingw64
505 MSYS rmills@rmillsmm-w10:~/temp/0.27-maintenance/build_mingw/bin $
clanmills commented 4 years ago

I've investigated @D4N 's caveat about cross building shared libraries. I'm pleased to say that it does successfully build, with the same restriction as the static libraries, which are: 1) You have to copy the 5 "system" dlls in ming_build_shared/bin 2) The suite passes and exhibits the same behaviour concern TZ and the new python tests.

clanmills commented 4 years ago

I've investigated using wine to run exiv2 from Fedora and it works! Both build_mingw and build_mingw_shared are working. As before you have to copy the 5 "system" DLLs into the bin.

Running the test harness is challenging because we don't have make and bash in the wine environment. I can think of two ways to get the test suite to run:

Either: Install MSYS2 into Wine

Or: Mount the Fedora build_mingw or build_mingw_shared directories on a Windows machine on which you have installed MSYS2.

There are notes in README.md about setting up MSYS2.

Once you are running MSYS2/bash, you will be able to run the test suite as documented in README.md

The line-ending issue with the python test suite has been fixed. #1146 #1150

clanmills commented 4 years ago

EUREKA!

Mounting the Fedora Directory in Windows works! I achieved that by getting Fedora to build into a shared drive on the Mac //Mac/Home/gnu/github/exiv2/0.27-maintenance/build_mingw_fedora. Build in the cross platform way using mingw-cmake.

Copy the 5 DLLs on Fedora to the bin:

$ for i in libexpat-1.dll libgcc_s_seh-1.dll libstdc++-6.dll libwinpthread-1.dll zlib1.dll ; do cp -v /usr/x86_64-w64-mingw32/sys-root/mingw/bin/$i bin ; done

Run the test suite in MSYS on Windows on the build:

$ cd //Mac/Home/gnu/github/exiv2/0.27/maintenance/build_mingw_fedora
$ export EXIV2_BINDIR=$pwd/bin
$ export EXIV2_EXT=.exe
$ cd ../test
$ make test

There are two issues: 1) The TZ puzzle: #485 I have submitted a fix for this. PR #1152 2) Three python tests fail for reasons that will not be investigated.

ERROR: test_run (bugfixes.redmine.test_issue_812.DoNotDestroyHardLinks)
FAIL: test_run (bugfixes.github.test_issue_1099.EmptyValueInCommandFile)
FAIL: test_run (bugfixes.redmine.test_issue_1054.Exiv2jsonRecursiveJsonTreeWithXMP)

I'm not going to work on the python test suite in this obscure environment. The JSON issue is another manifestation of the TZ issue. The other two are obscure.

The test suite works impressively well and I have 100% confidence that the build is good.

clanmills commented 4 years ago

1153