search

Exiv2 / exiv2

Image metadata library and tools
http://www.exiv2.org/
Other
911 stars 281 forks source link

0.27 : compilation is broken with NLS option enabled under MinGW #610

Closed cgilles closed 5 years ago

cgilles commented 5 years ago

compilation trace :

/mnt/devel/GIT/6.x/project/bundles/mxe/temp.build/ext_exiv2/ext_exiv2-prefix/src/ext_exiv2/src/futils.cpp: In function 'std::cxx11::string Exiv2::getProcessPath()': /mnt/devel/GIT/6.x/project/bundles/mxe/temp.build/ext_exiv2/ext_exiv2-prefix/src/ext_exiv2/src/futils.cpp:467:9: error: 'HANDLE' was not declared in this scope HANDLE processHandle = NULL; ^ /mnt/devel/GIT/6.x/project/bundles/mxe/temp.build/ext_exiv2/ext_exiv2-prefix/src/ext_exiv2/src/futils.cpp:468:9: error: 'processHandle' was not declared in this scope processHandle = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, FALSE, GetCurrentProcessId()); ^ /mnt/devel/GIT/6.x/project/bundles/mxe/temp.build/ext_exiv2/ext_exiv2-prefix/src/ext_exiv2/src/futils.cpp:468:37: error: 'PROCESS_QUERY_INFORMATION' was not declared in this scope processHandle = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, FALSE, GetCurrentProcessId()); ^ /mnt/devel/GIT/6.x/project/bundles/mxe/temp.build/ext_exiv2/ext_exiv2-prefix/src/ext_exiv2/src/futils.cpp:468:65: error: 'PROCESS_VM_READ' was not declared in this scope processHandle = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, FALSE, GetCurrentProcessId()); ^ /mnt/devel/GIT/6.x/project/bundles/mxe/temp.build/ext_exiv2/ext_exiv2-prefix/src/ext_exiv2/src/futils.cpp:468:82: error: 'FALSE' was not declared in this scope processHandle = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, FALSE, GetCurrentProcessId()); ^ /mnt/devel/GIT/6.x/project/bundles/mxe/temp.build/ext_exiv2/ext_exiv2-prefix/src/ext_exiv2/src/futils.cpp:468:109: error: 'GetCurrentProcessId' was not declared in this scope processHandle = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, FALSE, GetCurrentProcessId()); ^ /mnt/devel/GIT/6.x/project/bundles/mxe/temp.build/ext_exiv2/ext_exiv2-prefix/src/ext_exiv2/src/futils.cpp:468:110: error: 'OpenProcess' was not declared in this scope processHandle = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, FALSE, GetCurrentProcessId()); ^ /mnt/devel/GIT/6.x/project/bundles/mxe/temp.build/ext_exiv2/ext_exiv2-prefix/src/ext_exiv2/src/futils.cpp:470:13: error: 'TCHAR' was not declared in this scope TCHAR filename[MAX_PATH]; ^ /mnt/devel/GIT/6.x/project/bundles/mxe/temp.build/ext_exiv2/ext_exiv2-prefix/src/ext_exiv2/src/futils.cpp:471:58: error: 'filename' was not declared in this scope if (GetModuleFileNameEx(processHandle, NULL, filename, MAX_PATH) != 0) { ^ /mnt/devel/GIT/6.x/project/bundles/mxe/temp.build/ext_exiv2/ext_exiv2-prefix/src/ext_exiv2/src/futils.cpp:471:68: error: 'MAX_PATH' was not declared in this scope if (GetModuleFileNameEx(processHandle, NULL, filename, MAX_PATH) != 0) { ^ /mnt/devel/GIT/6.x/project/bundles/mxe/temp.build/ext_exiv2/ext_exiv2-prefix/src/ext_exiv2/src/futils.cpp:471:76: error: 'GetModuleFileNameEx' was not declared in this scope if (GetModuleFileNameEx(processHandle, NULL, filename, MAX_PATH) != 0) { ^ /mnt/devel/GIT/6.x/project/bundles/mxe/temp.build/ext_exiv2/ext_exiv2-prefix/src/ext_exiv2/src/futils.cpp:474:38: error: 'CloseHandle' was not declared in this scope CloseHandle(processHandle); ^ src/CMakeFiles/exiv2lib.dir/build.make:363 : la recette pour la cible « src/CMakeFiles/exiv2lib.dir/futils.cpp.obj » a échouée gmake[6]: [src/CMakeFiles/exiv2lib.dir/futils.cpp.obj] Erreur 1 gmake[6]: Attente des tâches non terminées.... /mnt/devel/GIT/6.x/project/bundles/mxe/temp.build/ext_exiv2/ext_exiv2-prefix/src/ext_exiv2/src/convert.cpp:1510:10: warning: 'bool {anonymous}::convertStringCharsetWindows(std::cxx11::string&, const char, const char)' defined but not used [-Wunused-function] bool convertStringCharsetWindows(std::string& str, const char from, const char to) ^ CMakeFiles/Makefile2:212 : la recette pour la cible « src/CMakeFiles/exiv2lib.dir/all » a échouée gmake[5]: [src/CMakeFiles/exiv2lib.dir/all] Erreur 2 Makefile:129 : la recette pour la cible « all » a échouée gmake[4]: [all] Erreur 2 ext_exiv2/CMakeFiles/ext_exiv2.dir/build.make:112 : la recette pour la cible « ext_exiv2/ext_exiv2-prefix/src/ext_exiv2-stamp/ext_exiv2-build » a échouée gmake[3]: [ext_exiv2/ext_exiv2-prefix/src/ext_exiv2-stamp/ext_exiv2-build] Erreur 2 CMakeFiles/Makefile2:415 : la recette pour la cible « ext_exiv2/CMakeFiles/ext_exiv2.dir/all » a échouée gmake[2]: [ext_exiv2/CMakeFiles/ext_exiv2.dir/all] Erreur 2 CMakeFiles/Makefile2:427 : la recette pour la cible « ext_exiv2/CMakeFiles/ext_exiv2.dir/rule » a échouée gmake[1]: [ext_exiv2/CMakeFiles/ext_exiv2.dir/rule] Erreur 2 Makefile:196 : la recette pour la cible « ext_exiv2 » a échouée gmake: [ext_exiv2] Erreur 2 FAILED COMMAND: ${MXE_BUILD_TARGETS}-cmake --build . --config RelWithDebInfo --target ext_exiv2 -- -j$CPU_CORES

[gilles@localhost mxe]$

In fact, in futils.cpp there are 2 problems in getProcessPath() : 

std::string getProcessPath()
{
    std::string ret("unknown");
#if defined(WIN32)                                       <======= Must be _MSC_VER
    HANDLE processHandle = NULL;
    processHandle = OpenProcess(PROCESS_QUERY_INFORMATION | PROCESS_VM_READ, FALSE, GetCurrentProcessId());
    if (processHandle != NULL) {
        TCHAR filename[MAX_PATH];
        if (GetModuleFileNameEx(processHandle, NULL, filename, MAX_PATH) != 0) {
            ret = filename;
        }
        CloseHandle(processHandle);
    }
#elif defined(__APPLE__)
    const int pid = getpid();
    char pathbuf[PROC_PIDPATHINFO_MAXSIZE];
    if (proc_pidpath (pid, pathbuf, sizeof(pathbuf)) > 0) {
        ret = pathbuf;
    }
#elif defined(__linux__) || defined(__CYGWIN__) || defined(__MINGW__)
    // http://stackoverflow.com/questions/606041/how-do-i-get-the-path-of-a-process-in-unix-linux
    char proc[100];
    char path[500];
    sprintf(proc,"/proc/%d/exe", getpid());
    ssize_t l = readlink (proc, path,sizeof(path)-1);     << =========== Do not exists under Mingw
    if (l>0) {
        path[l]=0;
        ret = path;
    }
#endif
#if defined(WIN32)
    const size_t idxLastSeparator = ret.find_last_of('\\');
#else
    const size_t idxLastSeparator = ret.find_last_of('/');
#endif
    return ret.substr(0, idxLastSeparator);
}

// ---------------------------------------

I take a look under Qt5 code which is fully portable and support very well MinGW, the readlink() call is wrapped with access()

Gilles Caulier

cgilles commented 5 years ago

If i replace :

ssize_t l = readlink (proc, path,sizeof(path)-1);

by : ssize_t l = 0; //readlink (proc, path,sizeof(path)-1);

All the core library compile fine.

[gilles@localhost build]$ make [ 20%] Built target xmp [ 48%] Built target exiv2lib_int Scanning dependencies of target exiv2lib [ 49%] Building CXX object src/CMakeFiles/exiv2lib.dir/futils.cpp.obj [ 50%] Building CXX object src/CMakeFiles/exiv2lib.dir/image.cpp.obj /mnt/devel/GIT/6.x/project/bundles/mxe/temp.build/ext_exiv2/ext_exiv2-prefix/src/ext_exiv2/src/image.cpp:871:77: warning: unused parameter 'useCurl' [-Wunused-parameter] BasicIo::AutoPtr ImageFactory::createIo(const std::wstring& wpath, bool useCurl) ^ [ 51%] Building CXX object src/CMakeFiles/exiv2lib.dir/ini.cpp.obj [ 52%] Building CXX object src/CMakeFiles/exiv2lib.dir/iptc.cpp.obj [ 54%] Building CXX object src/CMakeFiles/exiv2lib.dir/jp2image.cpp.obj [ 55%] Building CXX object src/CMakeFiles/exiv2lib.dir/jpgimage.cpp.obj [ 56%] Building CXX object src/CMakeFiles/exiv2lib.dir/matroskavideo.cpp.obj [ 57%] Building CXX object src/CMakeFiles/exiv2lib.dir/metadatum.cpp.obj [ 58%] Building CXX object src/CMakeFiles/exiv2lib.dir/mrwimage.cpp.obj [ 60%] Building CXX object src/CMakeFiles/exiv2lib.dir/orfimage.cpp.obj [ 61%] Building CXX object src/CMakeFiles/exiv2lib.dir/pgfimage.cpp.obj [ 62%] Building CXX object src/CMakeFiles/exiv2lib.dir/preview.cpp.obj [ 63%] Building CXX object src/CMakeFiles/exiv2lib.dir/properties.cpp.obj [ 64%] Building CXX object src/CMakeFiles/exiv2lib.dir/psdimage.cpp.obj [ 65%] Building CXX object src/CMakeFiles/exiv2lib.dir/quicktimevideo.cpp.obj [ 67%] Building CXX object src/CMakeFiles/exiv2lib.dir/rafimage.cpp.obj [ 68%] Building CXX object src/CMakeFiles/exiv2lib.dir/riffvideo.cpp.obj [ 69%] Building CXX object src/CMakeFiles/exiv2lib.dir/rw2image.cpp.obj [ 70%] Building CXX object src/CMakeFiles/exiv2lib.dir/tags.cpp.obj [ 71%] Building CXX object src/CMakeFiles/exiv2lib.dir/tgaimage.cpp.obj [ 72%] Building CXX object src/CMakeFiles/exiv2lib.dir/tiffimage.cpp.obj [ 74%] Building CXX object src/CMakeFiles/exiv2lib.dir/types.cpp.obj [ 75%] Building CXX object src/CMakeFiles/exiv2lib.dir/utilsvideo.cpp.obj [ 76%] Building CXX object src/CMakeFiles/exiv2lib.dir/value.cpp.obj [ 77%] Building CXX object src/CMakeFiles/exiv2lib.dir/version.cpp.obj [ 78%] Building CXX object src/CMakeFiles/exiv2lib.dir/webpimage.cpp.obj [ 80%] Building CXX object src/CMakeFiles/exiv2lib.dir/xmp.cpp.obj [ 81%] Building CXX object src/CMakeFiles/exiv2lib.dir/xmpsidecar.cpp.obj [ 82%] Building CXX object src/CMakeFiles/exiv2lib.dir/pngimage.cpp.obj [ 83%] Linking CXX shared library ../bin/libexiv2.dll [100%] Built target exiv2lib

So the problem is only located to found a right replacement of readlink() call under Mingw, or to disable this function for the moment under MinGW.

As i can see, readlink() support is problematic, as symbolic link is completly different under Windows.

https://stackoverflow.com/questions/39785170/replacement-for-readlink-function-in-windows

Gilles

cgilles commented 5 years ago

Patch to compile under Mingw with NLS support.

https://cgit.kde.org/digikam.git/tree/project/bundles/3rdparty/ext_exiv2/exiv2-0.27-mxe.patch

I use it with digiKam Windows bundle. Sound like the function getProcessPath() is only used with Exiv2 CLI tool, so we don't care as we use only the shared lib.

Gilles

clanmills commented 5 years ago

Seasons Greetings.

getProcessPath() was added about RC2 to assist the localisation code to find the pot files. I'm very surprised that it's creating trouble with MinGW. I believe it's only called from main() in exiv2.cpp.

From what you've said, this is inconvenient, but not preventing you from moving forward. The fix will be in Exiv2 v0.27.1 which I've scheduled for 31 March 2019.

cgilles commented 5 years ago

My little finger tells me that a specific #define must be set before to include unistd.h from mingw to be able to use readlink().

As i can see, the function exists well in mingw header provided by MXE. So we must be able to use it Gilles

cgilles commented 5 years ago

Look the unistd.h provided by MXE. Look especially the condition "#if __SSP_FORTIFY_LEVEL > 0" :

[gilles@localhost ssp]$ pwd /mnt/devel/GIT/6.x/project/bundles/mxe/build.win64/usr/lib/gcc/x86_64-w64-mingw32.shared/5.5.0/include/ssp

[gilles@localhost ssp]$ cat unistd.h /* Checking macros for unistd functions. Copyright (C) 2005, 2009 Free Software Foundation, Inc.

This file is part of GCC.

GCC is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3, or (at your option) any later version.

In addition to the permissions in the GNU General Public License, the Free Software Foundation gives you unlimited permission to link the compiled version of this file into combinations with other programs, and to distribute those combinations without any restriction coming from the use of this file. (The General Public License restrictions do apply in other respects; for example, they cover modification of the file, and distribution when not linked into a combine executable.)

GCC is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

Under Section 7 of GPL version 3, you are granted additional permissions described in the GCC Runtime Library Exception, version 3.1, as published by the Free Software Foundation.

You should have received a copy of the GNU General Public License and a copy of the GCC Runtime Library Exception along with this program; see the files COPYING3 and COPYING.RUNTIME respectively. If not, see http://www.gnu.org/licenses/. */

ifndef _SSP_UNISTD_H

define _SSP_UNISTD_H 1

include

include_next

if __SSP_FORTIFY_LEVEL > 0

undef read

undef readlink

undef getcwd

extern ssize_t SSP_REDIRECT (__read_alias, (int fd, void *__buf, size_t __nbytes), read);

extern inline attribute((always_inline__)) ssize_t read (int fd, void *buf, size_t nbytes) { if (ssp_bos0 (buf) != (size_t) -1 && nbytes > ssp_bos0 (buf)) chk_fail (); return __read_alias (fd, buf, __nbytes); }

extern int SSP_REDIRECT (__readlink_alias, (const char *restrict path, char *restrict __buf, size_t __len), readlink);

extern inline attribute((always_inline)) int readlink (const char *restrict path, char *restrict __buf, size_t len) { if (ssp_bos (__buf) != (size_t) -1 && len > ssp_bos (buf)) chk_fail (); return readlink_alias (path, buf, __len); }

extern char __SSP_REDIRECT (__getcwd_alias, (char __buf, size_t __size), getcwd);

extern inline attribute((always_inline)) char getcwd (char buf, size_t size) { if (ssp_bos (__buf) != (size_t) -1 && size > ssp_bos (buf)) chk_fail (); return getcwd_alias (buf, size); }

endif / __SSP_FORTIFY_LEVEL > 0 /

endif / _SSP_UNISTD_H /

clanmills commented 5 years ago

Thanks, Gilles. I'm very surprised by your report. Are you using MinGW/msys1.0 or MinGW/msys2?

I would like to have a break before digging into this. The effort to release v0.27 has been intense. Many 12 hour days.

cgilles commented 5 years ago

I don't install Msys or mingw directly. I use MXE cross compiler envirronnement for Linux. There is no windows, and i will never install this stuff in the future on my computer : it's not my religion (:=)))

So MXE do the stuff very : all the whole shared lib are compiled like this for the digiKam Windows installer : Qt5, opencv, boost, ffmpeg, KDElibs, etc...

So, i'm sure that a simple solution can be found for this problem...

Due to my hard work in my office to port a giant Java code to Qt5, i was very busy for Open Source cake while the week ends, since more than 2 months... So it's the holidays, and now i can look over the next world again...

I know well the time to pass to release open source. It's never complete. It still always the 5% to kill in the list. So, for me this kind of situation are normal. You must to process by priority the most important problem first, the rest it's for later (:=)))

Gilles

cgilles commented 5 years ago

Look like the path use to found unistd from mxe install : /mnt/devel/GIT/6.x/project/bundles/mxe/build.win64/usr/lib/gcc/x86_64-w64-mingw32.shared/5.5.0/include/ssp

It's G++ 5.5.0 so, it's probably MinGW 5.5.0 in background...

Gilles

clanmills commented 5 years ago

We've been building with GCC 7.3.0 and the MinGW build is fine. I'll look at 5.5.0.

Relax and take your dog for a walk.

cgilles commented 5 years ago

In all cases, the code below taken from getProcessPath() will never work under Windows through Mingw : /proc directory is Linux stuff, not Windows (:=)))...

elif defined(linux) || defined(CYGWIN) || defined(MINGW)

// http://stackoverflow.com/questions/606041/how-do-i-get-the-path-of-a-process-in-unix-linux
char proc[100];
char path[500];
sprintf(proc,"/proc/%d/exe", getpid());
ssize_t l = readlink (proc, path,sizeof(path)-1);     << =========== Do not exists under Mingw
if (l>0) {
    path[l]=0;
    ret = path;
}

endif

If getProcessPath() function is only used by exiv2 CLI tool, I recommend to move this function inside the Exiv2 CLI implementation, not n th shared library. At least, this last one will still compilable alone without problem.

Separate well both parts from Exiv2 is always a good idea, as you can disable the CLI at configuration time with cmake options.

Gilles

clanmills commented 5 years ago

I didn't write that code. I think Luis intends the WIN32 branch. It uses readlink on Linux (and possibly Cygwin). I think he cut'n'pasted the code from somewhere.

The code is working. I pulled down the MinGW64 build from exiv2.org. The command: $ exiv2 -vV --grep process --grep exec is reporting the correct information. I know because I added the code to version.cpp to report processpath by calling getProcessPath().

I'm building GCC 5.5.0 at the moment.

cgilles commented 5 years ago

The code was copied from stackoverflow... I see the link in source code...

Can you explain to me how this code work with readlink() and /proc/... under Windows ?

This path is specific to Linux. With a cross compilation, you generate a binary target which will run on Windows without anything about a Linux env. emulation installed (as Cygwin for ex).

Do you mean that Mingw layer reroute the internal Windows process list to a emulation layer somewhere in memory for Posix compatibility ? So the process list is dupplicate to a "virtual" FS in memory as Linux like ?

Gilles Caulier

clanmills commented 5 years ago

Windows manages processes in psapi (process api). Linux uses this /proc magic. I believe (on Cygwin at any rate), the /proc APIs are implemented in cygwin1.dll.

I couldn't build GCC5.5.0 from source. ./configure complains about dependencies which I believe I have installed.

I have installed GCC 5.4.0, however the compiler dies on the first file at psdk_inc/_fd_types.h complaining about conflicts with fd_set.

I will instrument getProcessPath() with some printf's to find out how/why this code is working.

clanmills commented 5 years ago

The code is executing the linux branch.

 std::string getProcessPath()
    {
        std::string ret("unknown");
    #if defined(WIN32)
    printf("in WIN32\n";
        HANDLE processHandle = NULL;
...
        }
    #elif defined(__APPLE__)
    printf("in __APPLE__\n");
...
    #elif defined(__linux__) || defined(__CYGWIN__) || defined(__MINGW__)
    printf("in __linux__\n");
        // http://stackoverflow.com/questions/606041/how-do-i-get-the-path-of-a-process-in-unix-linux
...
    #endif

When I run from the command-line:

514 MSYS64:rmills@rmillsmm-w7:~/gnu/github/exiv2/buildserver/build $ bin/exiv2 -vVg process --grep executable
in __linux__
exiv2 0.27.0
in __linux__
processpath=/c/msys64/home/rmills/gnu/github/exiv2/buildserver/build/bin
executable=C:\msys64\home\rmills\gnu\github\exiv2\buildserver\build\bin\exiv2.exe
515 MSYS64:rmills@rmillsmm-w7:~/gnu/github/exiv2/buildserver/build $

The comment about stack overflow is in version.cpp. I put it there about 6 years ago. Luis copy and pasted that code when he implemented getProcessPath() for use in localisation. The code in version.cpp reports the files in the Windows FS, so executable=c:\msys64... and Luis's getProcessPath() reports in the MinGW file-system so processpath=/c/msys64/home/rmills/gnu/github/exiv2/buildserver/build/bin

cygwin1.dll implements readlink(). You can see it with nm -g --demangle cygwin1.dll | grep readlink. MinGW/msys2 is derived from cygwin, however it loads different dlls:

513 MSYS64:rmills@rmillsmm-w7:~/gnu/github/exiv2/buildserver/build $ bin/exiv2 -vVg library
in __linux__
exiv2 0.27.0
in __linux__
library=C:\WINDOWS\SYSTEM32\ntdll.dll
library=C:\WINDOWS\System32\KERNEL32.DLL
library=C:\WINDOWS\System32\KERNELBASE.dll
library=C:\msys64\home\rmills\gnu\github\exiv2\buildserver\build\bin\msys-exiv2-27.dll
library=C:\msys64\usr\bin\msys-2.0.dll
library=C:\WINDOWS\System32\PSAPI.DLL
library=C:\WINDOWS\System32\WS2_32.dll
library=C:\WINDOWS\System32\RPCRT4.dll
library=C:\msys64\usr\bin\msys-gcc_s-seh-1.dll
library=C:\msys64\usr\bin\msys-stdc++-6.dll
library=C:\msys64\usr\bin\msys-intl-8.dll
library=C:\msys64\usr\bin\msys-iconv-2.dll
library=C:\msys64\usr\bin\msys-z.dll
library=C:\msys64\usr\bin\msys-expat-1.dll
library=C:\WINDOWS\System32\advapi32.dll
library=C:\WINDOWS\System32\msvcrt.dll
library=C:\WINDOWS\System32\sechost.dll
library=C:\WINDOWS\SYSTEM32\CRYPTBASE.DLL
library=C:\WINDOWS\System32\bcryptPrimitives.dll
514 MSYS64:rmills@rmillsmm-w7:~/gnu/github/exiv2/buildserver/build $

I think /usr/bin/msys-2.0.dll is something very like cygwin1.dll. It's a platform emulator that enables MinGW code to behave in a Linux like manner and the "transition" to the underlying Windows APIs happens there.

524 MSYS64:rmills@rmillsmm-w7:/usr/bin $ nm -g --demangle msys-2.0.dll | grep readlink
000000018011e5f0 T _sigfe_readlink
000000018011e5fe T _sigfe_readlinkat
00000001800e9b00 T readlink
000000018013e340 T readlinkat
525 MSYS64:rmills@rmillsmm-w7:/usr/bin $

The reason for the difference between version.cpp/dumpLibraryInfo() and futils.cpp/getProcessPath() is historical. I implemented dumpLibraryInfo() about 2012 and Luis implemented getProcessPath() a few weeks ago. Neither is better. Luis and I discussed the difference when I reviewed his PR for that change. #535

There's something different in your setup. It is unlikely that GCC 5.5 would deal with this in a fundamentally different way from GCC 7.3.0. I think you're using MinGW/msys1.0 which isn't supported by Exiv2 v0.27 because it was so troublesome. MinGW/msys2 works well.

Here's the output of uname -a on the two platforms:

MinGW/msys1/32: MINGW32_NT-6.2 RMILLSMBP-W7 1.0.18(0.48/3/2) 2012-11-21 22:34 i686 Msys
Msys2/64:       MSYS_NT-10.0 rmillsmm-w7 2.10.0(0.325/5/3) 2018-02-09 15:25 x86_64 Msys

This a very elderly 32 bit msys1.0 that is still on my laptop. The Msys2 is very recent (probably the current release). I believe it was installed in September 2018 when I started to work on releasing Exiv2 v0.27.

I know you like to build all the code that goes into DigiKam. Is it possible to Download the MinGW64 build from exiv2.org and just use it in DigiKam? Downloading and integrating our binary library might be a good work-around for this.

clanmills commented 5 years ago

Whoops. I've read your comments about using MXE to cross-compile. This issue is coming from that environment. I have never got cross-compilation to work on Exiv2.

I agree that this is probably quite easy to fix. I'll see if I can build Exiv2 on the elderly MinGW/msys1.0 on the laptop. If I can get that to work, we'll know the solution. However let me have a break for Christmas and the family.

Best Wishes to you and your family for Christmas and New Year. And thank you for the effort you put into Exiv2. Luis, Dan and I appreciate your feedback and insights.

clanmills commented 5 years ago

I've looked at building Exiv2 with msys/1.0. Impossible. I can't even build the dependencies expat and zlib. Further effort with msys/1.0 is almost certainly time wasted.

I've investigated cross-compiling. Perhaps this isn't so painful. Here's a good article about setting up Toolchain-cross-mingw32-linux.cmake

https://gitlab.kitware.com/cmake/community/wikis/doc/cmake/cross_compiling/Mingw

And once we have that (in <exiv2dir>/cmake/Toolchain-cross-mingw32-linux.cmake we should be able to build with:

$ cd <exiv2dir>
$ rm -rf build ; mkdir build ; cd build
$ cmake .. -DCMAKE_TOOLCHAIN_FILE=../cmake/Toolchain-mingw32.cmake -DCMAKE_INSTALL_PREFIX=$PWD/install
$ make

Here's Toolchain-cross-mingw32-linux.cmake

# the name of the target operating system
SET(CMAKE_SYSTEM_NAME Windows)

# Choose an appropriate compiler prefix

# for classical mingw32
# see http://www.mingw.org/
#set(COMPILER_PREFIX "i586-mingw32msvc")

# for 32 or 64 bits mingw-w64
# see http://mingw-w64.sourceforge.net/
set(COMPILER_PREFIX "i686-w64-mingw32")
#set(COMPILER_PREFIX "x86_64-w64-mingw32"

# which compilers to use for C and C++
find_program(CMAKE_RC_COMPILER NAMES ${COMPILER_PREFIX}-windres)
#SET(CMAKE_RC_COMPILER ${COMPILER_PREFIX}-windres)
find_program(CMAKE_C_COMPILER NAMES ${COMPILER_PREFIX}-gcc)
#SET(CMAKE_C_COMPILER ${COMPILER_PREFIX}-gcc)
find_program(CMAKE_CXX_COMPILER NAMES ${COMPILER_PREFIX}-g++)
#SET(CMAKE_CXX_COMPILER ${COMPILER_PREFIX}-g++)

# here is the target environment located
SET(USER_ROOT_PATH /home/erk/erk-win32-dev)
SET(CMAKE_FIND_ROOT_PATH  /usr/${COMPILER_PREFIX} ${USER_ROOT_PATH})

# adjust the default behaviour of the FIND_XXX() commands:
# search headers and libraries in the target environment, search 
# programs in the host environment
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

This all smells of msvc (resource compilers and 32 bit). However, I'll work on this after Christmas. I feel we could add "support for cross-compiling" in 0.27.1 In the process of getting cross-compiling from Linux to Windows to work, I will encounter the issues which Gilles has encountered from MXE.

piponazo commented 5 years ago

Hi guys. Thanks for reporting the issue and the investigation. I took a quick look to the issue but I cannot dedicate much time to this during the holidays. The problem comes from not treating MinGW as a first citizen in our CI system ... I'll investigate the way to add a MinGW job in Appveyor, test this functionality properly and fix the issue.

clanmills commented 5 years ago

We had a short discussion about deprecating features in 0.28 on Friday and we released v0.27 on Thursday. We need to do more work on Exiv2 v0.28 and decide what we should be deprecated and what we intend to do about deprecating features. Getting the compiler to flag deprecated classes in "the dots" sounds like a good idea in addition to putting this information into the release notes.

Jenkins does our MinGW builds on Windows using Msys2. @cgilles is using MXE to cross-build on Linux for MinGW. I've never used that and I'm investigating what's involved. I kicked off MXE to build on an Ubuntu 18.04 VM about 7am this morning. 12 hours later it's still running and has consumed 10GB. It appears to have pulled down about 200 libraries (mostly Qt) and is building them for Windows.

I've also tried to directly cross-compile using the Toolchain stuff mentioned above. I can't get libz to build because I haven't cross-compiled something (LibC, I think). I tried to cross compile "hello world" and that also complained about missing libraries. I'm hoping MXE will deliver something functional.

cgilles commented 5 years ago

You don't need to install whole MXE packages, but only few ones needs by Exiv2 to be crosscompiled, as G++ of course...

If you take a look to the script that i shared few week ago to cross compiled Exiv2 with MXE, all is listed. This will take only 30 mns to run. That all.

Gilles Caulier

clanmills commented 5 years ago

I'm not sure what script you are talking about. I'm pulling down this: https://github.com/KDE/digikam.git

This is a curious bug report. It cannot be reproduced on our supported build platforms. It takes 24 hours to build/install MXE, then we we need digicam (1.5 million lines of code). And we do all of this on Linux to reproduce an issue on Windows. I know you're playing games with me.

The scripts you mentioned. Is this them? https://github.com/Exiv2/exiv2/issues/473

-rwxr-xr-x 1 gilles gilles 3358 oct. 13 13:49 bootstrap-exiv2.mxe*
-rwxr-xr-x 1 gilles gilles 4992 oct. 13 13:51 build-mxe.sh*

I'll deal with this later in the week. Time to relax and have some fun with the family.

cgilles commented 5 years ago

Exaclty this script. No need more. With it you can cross compile Exiv2 as stand alone library for windows with the CLI tool if you turn on right cmake options.

That all. It's very simple and fast. You don't need to compile digiKam as you mention (:=)))...

Best

Gilles

Le lun. 24 déc. 2018 à 18:16, Robin Mills notifications@github.com a écrit :

I'm not sure what script you are talking about. I'm pulling down this: https://github.com/KDE/digikam.git

This is a curious bug report. It cannot be reproduced on our supported build platforms. It takes 24 hours to build/install MXE, then we we need digicam (1.5 million lines of code). And we do all of this on Linux to reproduce an issue on Windows. I know you're playing games with me.

The scripts you mentioned. Is this them? #473 https://github.com/Exiv2/exiv2/issues/473

-rwxr-xr-x 1 gilles gilles 3358 oct. 13 13:49 bootstrap-exiv2.mxe -rwxr-xr-x 1 gilles gilles 4992 oct. 13 13:51 build-mxe.sh

I'll deal with this later in the week. Time to relax and have some fun with the family.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Exiv2/exiv2/issues/610#issuecomment-449756525, or mute the thread https://github.com/notifications/unsubscribe-auth/ACd9eGoQYBkVIB6QPBZ6JxpOTiGdAileks5u8Qv5gaJpZM4ZfW15 .

kamgha commented 5 years ago

Hello,

I'm facing the same issue. I'm using a custom built FreeBSD-to-Windows GCC-MinGW64 cross-compiler (GCC 8.2.0 and a very recent MinGW64 from their repository). Trying to build a static Exiv2 without NLS (-DEXIV2_ENABLE_SHARED=OFF -DEXIV2_ENABLE_NLS=OFF).

It compiles fine with Exiv2 0.26, but the build process broken on 0.27.0.

clanmills commented 5 years ago

I've run build-mxe.sh (1 hour 9 minutes)

rmills@rmillsmm-ubuntu:/media/psf/Home/gnu/github/exiv2/exiv2$ chmod +x ./build-mxe.sh 
rmills@rmillsmm-ubuntu:/media/psf/Home/gnu/github/exiv2/exiv2$ ./build-mxe.sh 

** (gedit:9896): WARNING **: 08:08:46.568: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme ubuntu-mono-dark
build-mxe.sh : build a bundle MXE install with Exiv2 dependencies.
-----------------------------------------------------------------------
Target Windows architecture: 32 bits.
CPU Cores to use : 1
Cloning into '/media/psf/Home/gnu/github/exiv2/exiv2/build.win32'...

** (gedit:9896): WARNING **: 08:08:46.587: Could not load theme icon text-x-generic: Icon 'text-x-generic' not present in theme ubuntu-mono-dark
Checking out files: 100% (1135/1135), done.

---------- Updating MXE
Already up to date.

---------- Building Exiv2 low level dependencies with MXE
[create settings.mk]
[check reqs]
[download]    binutils-2.28.tar.bz2
[nonet lib]   /media/psf/Home/gnu/github/exiv2/exiv2/build.win32/usr/x86_64-pc-linux-gnu/lib/nonetwork.so                                                                         
[git-log]   1c77cfc0 qtdeclarative: fix build with gcc4.8
[build]       mxe-conf               i686-w64-mingw32.shared
[done]        mxe-conf               i686-w64-mingw32.shared                                  0 KiB          0m4.921s
[build]       binutils               i686-w64-mingw32.shared
[done]        binutils               i686-w64-mingw32.shared                                  0 KiB          3m0.245s
...
[build]       pthreads               i686-w64-mingw32.shared
[done]        pthreads               i686-w64-mingw32.shared                                  0 KiB          0m14.955s

Elaspsed time for script execution : 1 hours 9 minutes 26 seconds
rmills@rmillsmm-ubuntu:/media/psf/Home/gnu/github/exiv2/exiv2$

Then bootstrap-exiv2-mxe.sh. I had to read/edit the script to discover that it expects a full path to the build.win32 directory.

rmills@rmillsmm-ubuntu:/media/psf/Home/gnu/github/exiv2/exiv2$ ./bootstrap-exiv2-mxe.sh ${PWD}/build.win32
MXE target : 32 bits shared
Installing to /media/psf/Home/gnu/github/exiv2/exiv2/build.win32 for target i686-w64-mingw32.shared with build mode RelWithDebInfo and configure options -DBUILD_SHARED_LIBS=ON \
         -DEXIV2_ENABLE_XMP=ON \
         -DEXIV2_ENABLE_EXTERNAL_XMP=OFF \
         -DEXIV2_ENABLE_PNG=ON \
         -DEXIV2_ENABLE_NLS=ON \
         -DEXIV2_ENABLE_PRINTUCS2=ON \
         -DEXIV2_ENABLE_LENSDATA=ON \
         -DEXIV2_ENABLE_VIDEO=OFF \
         -DEXIV2_ENABLE_WEBREADY=OFF \
         -DEXIV2_ENABLE_DYNAMIC_RUNTIME=OFF \
         -DEXIV2_ENABLE_CURL=OFF \
         -DEXIV2_ENABLE_SSH=OFF \
         -DEXIV2_BUILD_SAMPLES=OFF \
         -DEXIV2_BUILD_PO=OFF \
         -DEXIV2_BUILD_EXIV2_COMMAND=ON \
         -DEXIV2_BUILD_UNIT_TESTS=OFF \
         -DEXIV2_BUILD_DOC=OFF \
         -DEXIV2_TEAM_EXTRA_WARNINGS=OFF \
         -DEXIV2_TEAM_WARNINGS_AS_ERRORS=OFF \
         -DEXIV2_TEAM_USE_SANITIZERS=OFF \
         -DEXIV2_ENABLE_WIN_UNICODE=OFF
MXE_BUILD_TARGETS = i686-w64-mingw32.shared
----- PATH ---------
/media/psf/Home/gnu/github/exiv2/exiv2/build.win32/usr/bin
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin
/usr/games
/usr/local/games
/snap/bin
-----------------------------
== Using MXE wrapper: /media/psf/Home/gnu/github/exiv2/exiv2/build.win32/usr/bin/i686-w64-mingw32.shared-cmake
     - cmake version 3.10.0
     - warnings for unused CMAKE_POLICY_DEFAULT variables can be ignored
== Using MXE toolchain: /media/psf/Home/gnu/github/exiv2/exiv2/build.win32/usr/i686-w64-mingw32.shared/share/cmake/mxe-conf.cmake
== Using MXE runresult: /media/psf/Home/gnu/github/exiv2/exiv2/build.win32/usr/share/cmake/modules/TryRunResults.cmake
loading initial cache file /media/psf/Home/gnu/github/exiv2/exiv2/build.win32/usr/share/cmake/modules/TryRunResults.cmake
-- The CXX compiler identification is GNU 5.5.0
-- The C compiler identification is GNU 5.5.0
...

Then I run make. It has been downloading and building for several hours.

rmills@rmillsmm-ubuntu:/media/psf/Home/gnu/github/exiv2/exiv2/build.win32$ make
[build]       mxe-conf               i686-w64-mingw32.static
[done]        mxe-conf               i686-w64-mingw32.static                                  0 KiB          0m4.677s
[build]       binutils               i686-w64-mingw32.static
[done]        binutils               i686-w64-mingw32.static                                  0 KiB          2m50.916s
[build]       mingw-w64              i686-w64-mingw32.static
[done]        mingw-w64              i686-w64-mingw32.static                                  0 KiB          0m4.961s
[build]       gcc                    i686-w64-mingw32.static
[done]        gcc                    i686-w64-mingw32.static                                  0 KiB          31m19.560s
....

I hope it will finish today and I'll investigate the issue you're reported.

I expect those tools will work fine, once they are correctly installed and in working order.

clanmills commented 5 years ago

@kamgha I have never succeeded in getting anything to cross-compile. This isn't about Exiv2. I have never even succeeded in getting hello.cpp to build. @cgilles advocates MXE as solid, fast and reliable, I am still to succeed on building in this environment.

We had a bug reported on GCC 8.2.0 in September. I was able to build static, however I could not build shared. I don't remember the setting of NLS. I'm confident that you can build with/without NLS.

The MinGW build is documented in README.md and works on Windows. If you don't have Windows, Microsoft provide developments VMs which are valid for 90 days. I've successfully built on that platform.

clanmills commented 5 years ago

After another few hours of building, MXE dropped dead having not compiled a single line of Exiv2 code.

Makefile:796: recipe for target '/media/psf/Home/gnu/github/exiv2/exiv2/build.win32/usr/i686-w64-mingw32.static/installed/nettle' failed
make: *** [/media/psf/Home/gnu/github/exiv2/exiv2/build.win32/usr/i686-w64-mingw32.static/installed/nettle] Error 1
rmills@rmillsmm-ubuntu:/media/psf/Home/gnu/github/exiv2/exiv2/build.win32$ cmake --build . --target help
Error: could not load cache
rmills@rmillsmm-ubuntu:/media/psf/Home/gnu/github/exiv2/exiv2/build.win32$ cmake --build .
Error: could not load cache
rmills@rmillsmm-ubuntu:/media/psf/Home/gnu/github/exiv2/exiv2/build.win32$ make
[build]       nettle                 i686-w64-mingw32.static
[done]        nettle                 i686-w64-mingw32.static                                  0 KiB          0m34.772s
[download]    gnutls-3.6.5.tar.xz
[build]       gnutls                 i686-w64-mingw32.static

Failed to build package gnutls for target i686-w64-mingw32.static!
------------------------------------------------------------
guile/src/Makefile.am:117: warning: '%'-style pattern rules are a GNU make extension
configure: WARNING: unrecognized options: --enable-gtk-doc, --enable-gtk-doc-html, --enable-gtk-doc-pdf, --with-html-dir, --disable-doxygen
configure: error: pwd does not report name of working directory
Makefile:796: recipe for target 'build-only-gnutls_i686-w64-mingw32.static' failed
make[1]: *** [build-only-gnutls_i686-w64-mingw32.static] Error 1
make[1]: Leaving directory '/media/psf/Home/gnu/github/exiv2/exiv2/build.win32'
real    0m50.163s
user    0m32.980s
sys 0m6.118s
------------------------------------------------------------
[log]      /media/psf/Home/gnu/github/exiv2/exiv2/build.win32/log/gnutls_i686-w64-mingw32.static

Makefile:796: recipe for target '/media/psf/Home/gnu/github/exiv2/exiv2/build.win32/usr/i686-w64-mingw32.static/installed/gnutls' failed
make: *** [/media/psf/Home/gnu/github/exiv2/exiv2/build.win32/usr/i686-w64-mingw32.static/installed/gnutls] Error 1
rmills@rmillsmm-ubuntu:/media/psf/Home/gnu/github/exiv2/exiv2/build.win32$

I don't intend to spend further time on this issue. The supported method of building Exiv2 for MinGW is to use msys2 as documented in README.md.

clanmills commented 5 years ago

@kamgha I've been able to get onto your ssh server. That's very helpful. I'm going to have my breakfast (it's 09:15 in England) and then work on your server.

@cgilles is a very helpful and frequently contributes to Exiv2. He's a very productive and effective engineer. I have no doubt that MXE delivers happiness for him. I'm having a different and very frustrating experience.

cgilles commented 5 years ago

...and i don't understand why... you arrive to this conclusion... With the 2 scripts that provide, you just run

That all. It's simple and fast. no more to do, excepted to run make in build directory crated by cmake and enjoy the compilation bug, if one exists.

I don't understand why you try to re-invent the wheel each time...

Voilà. I use MXE on a huge collection of codes and projects (open source or not), all configured with Cmake, and it's work as expected. Of course, i will discover some dysfunctions when i update some components to a major version, but nothing really complex to solve.

For me cross compiling is the ultimate solution, especially for Windows. I don't want to use the Windows stuff to compile and link, and particuliary the M$ compiler. It's THE waste of time for me. All is hosted on the same computer running Linux. No Windows, expected one to check install and runtime to to capture backtraces with DrMinGW. If i want to hack, i use QtCreator and the GDB provided inside. If i want to quickly fix a bug and test : I share the build directory between the Windows (running in virtual box) and the Linux host where i cross compile. I fix the code under Linux, i cross compile, and i run finaly the executable shared by Linux under the Windows target.

As 80% of 1.5M of C++ code from digiKam is based on Qt, the code is mostly the same for all platforms. No need to recreate the background implementation for each platform. It's a waste of time and already done with Qt5 framework. This speedup development and limit the low level bugs. I can concentrate on real application fixes and forget all the rest.

Gilles Caulier

clanmills commented 5 years ago

@cgilles Our new friend @kamgha has provided me with SSH access to a Linux machine and I've reproduced his troubles with src/bascio.cpp and src/futils.cpp I'm optimistic that we'll get this fixed shortly.

I know you like MXE, CMake and other cross-platform tools. I'm doing my best to support you. We'll get there. We always do! You do things your way, and so do I. It's my sincere objective to get everything working for everybody!

clanmills commented 5 years ago

I've had several emails and Skype discussions with @kamgha about this. I've found his build.sh script on his ssh server to be very useful and easy to understand. I've crafted my own version of his script which I've included in a tar-ball along with the 64 bit DLLs required to enable the build run on wine. I've also included a patch for src/futils.cpp. These files should be added as the directory contrib/mingw-cross. There is no need for a toolchain file.

My goal is to ensure that cross-compiling on Linux for MinGW works and will be supported in v0.27.1. I hope to build/publish such builds from Jenkins.

The preferred alternative to the script is to build dependencies with conan. I haven't investigate that yet. So, I'm posting the script (and dlls and patch) as a work in progress effort. This matter will receive more attention before Exiv2 v0.27.1 which is scheduled for 31 March 2019.

@cgilles I appreciate your enthusiasm for MXE. Regrettably, I found it frustrating and did not work. However I believe MXE is a superset of the prerequisites I have documented in build.sh in the attached archive. So, you have a different way to drive the MinGW-crossplatform tools. However we share the destination and are using identical tools to reach our goal.

I am very appreciative of the time put into this issue by @kamgha. For sure there will be a "proper" fix for futils.cpp in Exiv2 v0.27.1. I'm puzzled that the issue reported by both you and Gilles concerning basicio.cpp has not surfaced on my machines. When it does, I am confident we will cooperate to solve and fix the issue. Thank You for your help. mingw-cross.tar.gz

clanmills commented 5 years ago

Very pleased to say that using conan to build the dependencies caused no pain at all.

I've written this script conan-build.sh to work with the archive above.

We'll need to on the "proper" fix for futils.cpp which is currently broken for other reasons (#615). I'll get this totally fixed and documented for v0.27.1

#!/usr/bin/env bash

echo ------------------------------
echo PWD = $PWD
echo this = $0

EXIV2=$(realpath ../..)
if [ ! -e $EXIV2/CMakeLists.txt ]; then 
   echo "***" file $EXIV2/CMakeLists.txt does not exist "***"
   exit 1
fi

# Save the profile in a file
cat - > conan-build.profile <<EOF
\$toolchain=/usr/bin
target_host=x86_64-w64-mingw32
cc_compiler=gcc
cxx_compiler=g++

[env]
CONAN_CMAKE_FIND_ROOT_PATH=$toolchain
CHOST=\$target_host
AR=\$target_host-ar
AS=\$target_host-as
RANLIB=\$target_host-ranlib
CC=\$target_host-$cc_compiler
CXX=\$target_host-$cxx_compiler
STRIP=\$target_host-strip
RC=\$target_host-windres

[settings]
# We are building in Ubuntu Linux
os_build=Linux
arch_build=x86_64

# We are cross building to Windows
os=Windows
arch=x86_64
compiler=gcc

# Adjust to the gcc version of your MinGW package
compiler.version=7.3
compiler.libcxx=libstdc++11
build_type=Release

EOF

# request conan to build/download/install dependencies
conan install ../.. --build missing --profile conan-build.profile

# set up CMake cross compiler (see build.sh)
COMPILER_PREFIX="x86_64-w64-mingw32"

CM_GENERAL="-DCMAKE_SYSTEM_NAME=Windows                 \
-DCMAKE_C_COMPILER=$(which ${COMPILER_PREFIX}-gcc)      \
-DCMAKE_CXX_COMPILER=$(which ${COMPILER_PREFIX}-g++)    \
-DCMAKE_RC_COMPILER=$(which ${COMPILER_PREFIX}-windres) \
-DCMAKE_MAKE_PROGRAM=$(which make)                      \
-DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=NEVER               \
-DCMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY                \
-DCMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY                \
-DCMAKE_BUILD_TYPE=Release"

# -DCMAKE_FIND_ROOT_PATH="/usr/${COMPILER_PREFIX}"      \

# build exiv2
echo ---- building exiv2 in $PWD -------------
cmake    ../.. $CM_GENERAL                             \
         -DBUILD_SHARED_LIBS=0                         \
         -DEXIV2_ENABLE_DYNAMIC_RUNTIME=Off            \
         -DCMAKE_INSTALL_PREFIX=$PWD

cmake    --build . --config Release
cp       gcc64/*.dll     bin/
wine     bin/exiv2 -vV

# That's all Folks!
##