Windows Build Issue

[chhil]

I set up the build environment from scratch.

Couple of things that were different

SAXON : The requirement to download it and place it and set classpath, uses saxon9he.jar but thats no longer available and I used SaxonHE10-2J.zip , extracted the files correctly and updated classpath to use saxon-he-10.2.jar.

Older version of lensfun is no longer available so I went with what got installed and did the lensfun update.

---

While trying to build I got a

-- The following OPTIONAL packages have not been found:

 * Saxon

Then later I got the error

[  6%] Checking validity of data/darktableconfig.xml
[  6%] Built target validate_darktableconfig_xml
[  6%] Built target generate_conf
[  6%] Built target generate_preferences
[  6%] Built target validate-cameras.xml
[ 18%] Built target rawspeed
[ 18%] Building C object src/CMakeFiles/lib_darktable.dir/common/camera_control.c.obj
C:/msys64/home/chill/darktable/src/common/camera_control.c: In function 'dt_camctl_import':
C:/msys64/home/chill/darktable/src/common/camera_control.c:1044:73: error: passing argument 3 of 'gp_file_get_data_and_size' from incompatible pointer type [-Werror=incompatible-pointer-types]
 1044 |       if((res = gp_file_get_data_and_size(camfile, (const char**)&data, &size)) < GP_OK)
      |                                                                         ^~~~~
      |                                                                         |
      |                                                                         size_t * {aka long long unsigned int *}
In file included from C:/msys64/mingw64/include/gphoto2/gphoto2.h:50,
                 from C:/msys64/home/chill/darktable/src/common/camera_control.h:30,
                 from C:/msys64/home/chill/darktable/src/common/camera_control.c:21:
C:/msys64/mingw64/include/gphoto2/gphoto2-file.h:136:30: note: expected 'long unsigned int *' but argument is of type 'size_t *' {aka 'long long unsigned int *'}
  136 |           unsigned long int *size);
      |           ~~~~~~~~~~~~~~~~~~~^~~~
cc1.exe: all warnings being treated as errors
make[2]: *** [src/CMakeFiles/lib_darktable.dir/build.make:1831: src/CMakeFiles/lib_darktable.dir/common/camera_control.c.obj] Error 1
make[1]: *** [CMakeFiles/Makefile2:2331: src/CMakeFiles/lib_darktable.dir/all] Error 2
make: *** [Makefile:172: all] Error 2

[ptilopteri]

-- The following OPTIONAL packages have not been found:

• Saxon

this does not only affect the windows build

I have tried saxon{6,8,9.10} on openSUSE Tumbleweed and non satisfy the build script.

[jenshannoschwalm]

Saxon seems not to be maintained any more

[chhil]

So we can rule out saxon.

What about the compile error?

[parafin]

Which version of libgphoto do you have?

[chhil]

chill@DESKTOP-XXX MINGW64 ~
$ echo $CAMLIBS
/mingw64/lib/libgphoto2/2.5.23/

chill@DESKTOP-XXX MINGW64 ~
$ echo $IOLIBS
/mingw64/lib/libgphoto2_port/0.12.0/

I verified the existence of these paths when I had to set the export variable in my .bash_profile file.

export CAMLIBS="/mingw64/lib/libgphoto2/2.5.23/"
export IOLIBS="/mingw64/lib/libgphoto2_port/0.12.0/"

[chhil]

@jenshannoschwalm Looks like this

unsigned long int size; was changed to size_t size;

This would most likely cause the error shown in the OP.

You can view the diff here. https://github.com/darktable-org/darktable/commit/97b7be694c636ed434b28531da1fcf825d88783d#diff-f3144dc7749fc42450fde34372cd4193

[chhil]

To make sure it wasn't a build environment issue, I got a previous version to the camera_control.c and replaced the existing one. No longer got that compile error but I got the error that had made me create a new build environment (discussed in https://github.com/darktable-org/darktable/issues/6459#issuecomment-704901629).

[ 49%] Generating authors.h for about dialog.
/bin/bash: C:/msys64/home/chill/darktable/tools/authors_h.sh: No such file or directory
make[2]: *** [src/libs/CMakeFiles/generate_authors_h.dir/build.make:85: src/libs/tools/darktable_authors.h] Error 127
make[1]: *** [CMakeFiles/Makefile2:3522: src/libs/CMakeFiles/generate_authors_h.dir/all] Error 2
make: *** [Makefile:172: all] Error 2

---

As I see it there are 2 issues,

1. camera_control.c
2. Problem with the "bash" inclusion in the src/libs/CMakeLists.txt b/src/libs/CMakeLists.txt . Although there are others that can build, I have a clean ,just setup build environment so I don't think it's a user error. Can someone check the windows build.tx to make sure the variables and paths are correctly mentioned based on current msy2/mingw installs? https://github.com/darktable-org/darktable/blob/master/packaging/windows/BUILD.txt#L74-L90

[chhil]

To work around the error caused by the introduction of bash https://github.com/darktable-org/darktable/issues/6459#issuecomment-704270870 I changed it to /usr/bin/bash and it passed the problem area.

It was close to being built but failed when it could not find lensfun \version_1 (folder version_2 exists for me, something that got created following the build instructions).

[100%] Built target manpages
Run CPack packaging tool...
CPack: Create package using NSIS
CPack: Install projects
CPack: - Run preinstall target for: darktable
CPack: - Install project: darktable []
CPack: -   Install component: DTApplication
CMake Error at C:/msys64/home/chill/darktable/build/src/cmake_install.cmake:154 (file):
  file INSTALL cannot find
  "C:/msys64/mingw64/bin/../share/lensfun/version_1": No error.
Call Stack (most recent call first):
  C:/msys64/home/chill/darktable/build/cmake_install.cmake:38 (include)

So, I copied version_2 to version_1.

This resulted in DT eventually building. I don't know if there will be functional problems with this build, but it took tweaking to get it to build. So any new user or an exisiting user tries to build with the current msys/mingw install, it will not build in all likelihood.

---

Please don't close this as the issues listed will prevent the user from getting a window's build.

[parafin]

Renaming version_2 to version_1 is not a correct fix (fixing cmake is), I expect lens correction plugin not to work correctly after that. If you don't care about lens correction, you can build without lensfun altogether BTW (-DUSE_LENSFUN=OFF).

[chhil]

I need the lens correction, very useful for images shot using wide angle glass. I will hope the devs can take a look at cmake scripts.

[phrrk]

@chhil did you follow all the instructions in the darktable/packaging/windows/build.txt below? They work for me. I vaguely recall having a similar issue to you when I forgot the moving of files to python3.8.

NOTE: at the moment, the lensfun version provided by MSYS2 (0.3.95-1) is broken. If you get errors in the build process, manually downgrade lensfun to version 0.3.2-4: Download the file mingw-w64-x86_64-lensfun-0.3.2-4-any.pkg.tar.xz from http://repo.msys2.org/mingw/x86_64/ Put the file in your MSYS2 home folder, then from MSYS2 terminal: $ pacman -U mingw-w64-x86_64-lensfun-0.3.2-4-any.pkg.tar.xz Also in order to prevent pacman from updating the lensfun package, open the file /etc/pacman.conf, find the line:

Pacman won’t upgrade packages listed in IgnorePkg and members of IgnoreGroup

and below, add the following line
IgnorePkg = mingw-w64-x86_64-lensfun
Copy the lensfun folder and the lensfun-0.3.2-py3.6.egg-info file from /mingw64/lib/python3.6/site-packages to /mingw64/lib/python3.8/site-packages.

From MINGW64 terminal, update your lensfun database: $ lensfun-update-data

[chhil]

Yes I followed the instructions. Did it multiple times. I have been building darktable for quite some time now. Old Lensfun package is not available anymore, you can verify it by following the url you shared I have listed what's no longer valid from the build.txt in the comments.

[phrrk]

Ok I see. I guess that the best thing would be to see if we can get the old version 0.3.2 package reinstalled in the repository since version 0.3.95 is broken and of no use to anyone. In the meantime I do have a copy of mingw-w64-x86_64-lensfun-0.3.2-4-any.pkg.tar.xz that I can let you have if you wish. Also going back to Saxon version 6.5.5 is still available (see http://saxon.sourceforge.net/ near the end of the page, and I have just checked the link still exists). This works for me with docbook-xsl-saxon-1.00. Reading around it should be possible to use later versions of Saxon by forcing use of the XSLT 1.0 processor.

[chhil]

@phrrk

In the meantime I do have a copy of mingw-w64-x86_64-lensfun-0.3.2-4-any.pkg.tar.xz

Please do share. There are distros that have it but not knowing how different they could be, I would rather use a copy that I know has worked.

As for saxon... From the build documentation...

   # Update your CLASSPATH
    CLASSPATH=$CLASSPATH:~/java/saxon/saxon9he.jar:~/java/docbook-xsl-saxon/saxon65.jar

saxon65.jar is already available via dockbook-xsl, the saxon9he.jar is not available and hence I used 10.2he. ("9he" is available through mavencentral but I am not sure if I should use that.

When the saxon url is provided for download, mentioning the version number would help the user. For a person that is not familiar, the classpath setting to the 9he jar would be meaningless and will not relate that the jar is the one that was downloaded.

[phrrk]

@chhil I have raised the lensfun issue here https://github.com/msys2/MINGW-packages/issues/7108 and a positive outcome and a positive outcome looks likely. If not I will post my copy of the package.

On Saxon I cleared out everything relevant on my system and started afresh with eventually a successful outcome, i.e. I could produce copies of the manual. This what I did.

1. Download the latest version of saxonhe from https://sourceforge.net/projects/saxon/files/Saxon-HE/, i.e. SaxonHE10-2J.zip
2. Extract this into directory ~/java/saxon10 on my msys2 system
3. Change name of ~/java/saxon10/saxon-he-10.2.jar to ~/java/saxon10/saxon.jar
4. Download the file docbook-xsl-saxon-1.00.zip from https://sourceforge.net/projects/docbook/files/docbook-xsl-saxon/1.00/
5. Extract zip file, go in the folder docbook-xsl-saxon-1.00 Copy files to ~/java/docbook-xsl-saxon

6. Make following additions to .bash_profile

   CLASSPATH=$CLASSPATH: \~/java/saxon10/saxon.jar:\~/java/docbook-xsl-saxon/saxon65.jar
   export CLASSPATH export SAXON_INSTALL_DIR=\~/java/saxon10

item 3 seems to be essential for saxon to be found during run of cmake, i.e. it looks for saxon.jar I suspect this is why @chhil encountered problems

I agree with you that the current build.txt file needs changing and if the above works for you (@chill) I will submit an appropriate PR.

[chhil]

Thank you @phrrk, appreciate your follow ups.

Copied the lensfun 0.3.2-4 package from https://github.com/msys2/MINGW-packages/files/5365977/lensfun-0.3.2-4.zip. Unzipped it and copied file mingw-w64-i686-lensfun-0.3.2-4-any.pkg.tar.xz to my home directory C:\msys64\home\chill\

Opened a msys2 shell and ran pacman -U mingw-w64-i686-lensfun-0.3.2-4-any.pkg.tar.xz It got installed.

Added the ignore package to prevent unintentional upgrade of lensfun as documented.

Ran the lensfun-update-data and a version_1 folder was created.

Then the following one doesn't hold true as we no longer use python 3.6. There is only a python 3.8. So skipped this part. https://github.com/darktable-org/darktable/blob/master/packaging/windows/BUILD.txt#L38

Configured the classpath like yours. Since I copied it straight from the comment, I had to remove a space between the first ":" and "~" from CLASSPATH=$CLASSPATH: ~/java/saxon10/saxon.jar:~/java/docbook-xsl-saxon/saxon65.jar

# Update your CLASSPATH
CLASSPATH=$CLASSPATH:~/java/saxon/saxon.jar:~/java/docbook-xsl-saxon/saxon65.jar
export CLASSPATH
export SAXON_INSTALL_DIR=~/java/saxon

This fixed my issue with optional package saxon being found now.

The user manual are building fine too. On a side not is there a way to just build the default English user manual?

 For building the usermanual either add -DBUILD_USERMANUAL=ON flag to the first cmake command line, or if you want to build only the usermanual (and not the application) use:
        $ cmake --build . --target usermanual

-DBUILD_USERMANUAL=ON worked for me while cmake --build . --target usermanual did not work, gave me an error

$ cmake --build . --target usermanual
Error: could not load cache

The build shell script I run is

#!/bin/bash
cd ~
#rm -R ./darktable/
#git clone git://github.com/darktable-org/darktable.git

cd darktable
#git pull
#git checkout 580bf49
git submodule init
git submodule update
mkdir build

cd build
cmake -G "MSYS Makefiles" -DCMAKE_BUILD_TYPE=RelWithDebInfo -
# to build manual
# cmake -G "MSYS Makefiles" -DBUILD_USERMANUAL=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo -
DCMAKE_INSTALL_PREFIX=./debugReleaseWithDebugDT ../.
cmake --build .
cmake --build . --target install
cmake --build . --target package

There were 17 warnings while building the user manual. I can't tell the importance of them. Maybe someone can look into them too. [Uploading Output Building usermanual.txt…]()

[phrrk]

cmake --build . --target darktable-usermanual

from inside build directory works for me to build only English version of manual (having previously run cmake to set up Makefiles with in my case cmake -G "MSYS Makefiles" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/opt/darktable ../.)

[chhil]

The only issue that prevents a successful build is the fix for https://github.com/darktable-org/darktable/issues/6459#issuecomment-706577534

[phrrk]

I thought I would summarise where I think we have got to. There are 3 issues currently builing under windows;

1. Saxon and building the manuals - solved I think, but BUILD.txt needs to be updated

2. version 0.3.2 of lensfun no longer being available - I convinced the MSYS2 people to revert from 0.3.95, but that seems to have run into a bit of a problem see: https://github.com/msys2/MINGW-packages/issues/7108 presumably that will get solved soon

3. The use of bash, sh or whatever in .../src/libs/MakeLists.txt. It seems full path to bash (/usr/bin/bash or /bin/bash) works for everyone. sh works for me, but not bash (despite them being one and the same). No shell at all being given works for some. Which all seems very strange.

@codingdave I tried your stripped version of the offending bit of the Makefile generated by CMakeLists.txt

$ LANG=C LC_ALL=C bash C:/msys64/home/jim/darktable/tools/authors_h.sh C:/msys64/home/jim/darktable/AUTHORS C:/msys64/home/jim/darktable/jim/src/libs/tools/darktable_authors.h

is ok for me, but cmake -E env LANG=C LC_ALL=C bash C:/msys64/home/jim/darktable/tools/authors_h.sh C:/msys64/home/jim/darktable/AUTHORS C:/msys64/home/jim/darktable/jim/src/libs/tools/darktable_authors.h

which is how it appears in the relevant Makefile gives the usual error

/bin/bash: C:/msys64/home/jim/darktable/tools/authors_h.sh: No such file or directory

replacing bash by sh in the above

cmake -E env LANG=C LC_ALL=C sh C:/msys64/home/jim/darktable/tools/authors_h.sh C:/msys64/home/jim/darktable/AUTHORS C:/msys64/home/jim/darktable/jim/src/libs/tools/darktable_authors.h

works. So it looks like the cmake introduces the problems we encounter. Are we all on the latest version of cmake (3.17.3)?

[chhil]

I am on cmake 3.17.3

$ cmake -version
cmake version 3.17.3

[parafin]

There's probably a difference between starting bash directly from bash or from cmake, and between starting bash as bash and as sh.

How about replacing bash with /bin/bash or /usr/bin/bash? Does it behave differently? What does cmake -E env env output?

[codingdave]

Thanks for summarizing and organizing @phrrk.

$ cmake --version
cmake version 3.17.3

1: not working on my side but this is optional afaik so I don't care right now 2: Im currently running http://repo.msys2.org/mingw/x86_64/mingw-w64-x86_64-lensfun-0.3.2-4-any.pkg.tar.xz, which leads to another issue but I dont care right now because 3: Im working on fixing this before looking at the others.

@parafin I guess you are right. https://cmake.org/cmake/help/v3.17/generator/MSYS%20Makefiles.html#generator:MSYS%20Makefiles indicates we should not use bash but sh instead, more specifically it seems like always /bin/sh is used when using MSYS generated makefiles, so we should stick to that. That brings me to question

Q1: What is the reason for bash? Q2: Also why are we changing the environment to LC_ALL=C and LANG=C? we could do this in authors_h.sh instead.

$ cat foo_sh.sh
#!/bin/sh
echo foo sh

$ ./foo_sh.sh
foo sh

$ cat foo_bash.sh
#!/bin/bash
echo foo bash

$ ./foo_bash.sh
foo bash

$ /bin/sh --version
GNU bash, version 4.4.23(2)-release (x86_64-pc-msys)

$ /bin/bash --version
GNU bash, version 4.4.23(2)-release (x86_64-pc-msys)

$ which sh
/usr/bin/sh

$ /usr/bin/sh --version
GNU bash, version 4.4.23(2)-release (x86_64-pc-msys)

$ which bash
/usr/bin/bash

$ /usr/bin/bash --version
GNU bash, version 4.4.23(2)-release (x86_64-pc-msys)

$ env | grep "^PATH="
PATH=/mingw64/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/C/Program Files/Java/jre1.8.0_261/bin:/home/david/java/fop

$ cmake -E ./foo_sh.sh
CMake Error: cmake version 3.17.3

$ cmake -E ./foo_bash.sh
CMake Error: cmake version 3.17.3

$ cmake -E env env |grep "^PATH="
PATH=/mingw64/bin:/usr/local/bin:/usr/bin:/usr/bin:/opt/bin:/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/c/Program Files/Java/jre1.8.0_261/bin:/home/david/java/fop

So the difference between the pathes is for the cmake environment: missing /bin, removed backslash, lowercased drive letter:

msys:  /bin:
cmake: /usr/bin:

msys:  /c/Windows/System32/WindowsPowerShell/v1.0/
cmake: /c/Windows/System32/WindowsPowerShell/v1.0

msys:  /C/Program Files/Java/jre1.8.0_261/bin
cmake: /c/Program Files/Java/jre1.8.0_261/bin

edit Doesn' t matter what I pass along to cmake -E env I cannot start a script directly.

$ cmake -E env WHAT=ever ./foo_sh.sh                                                                                                                           
%1 is not a valid Win32 application

Any executable is required, though with -E

$ cmake -E ./foo_sh.sh
CMake Error: cmake version 3.17.3

$ cmake -E env WHAT=ever bash ./foo_sh.sh
foo sh

$ cmake -E env WHAT=ever sh -c "which sh"
/usr/bin/sh

cmake -E env WHAT=ever sh -c "which bash"
/usr/bin/bash

So I think -E is not the correct way to go.

[codingdave]

Trying to pass the MSYS path to CMAKE fails:

$ cmake -E env PATH=/mingw64/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/C/Program\ Files/Java/jre1.8.0_261/bin:/home/david/java/fop echo $PATH
C:\msys64\mingw64\bin;C:\msys64\usr\local\bin;C:\msys64\usr\bin;C:\msys64\usr\bin;C:\msys64\opt\bin;C:\Windows\System32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\msys64\usr\bin\site_perl;C:\msys64\usr\bin\vendor_perl;C:\msys64\usr\bin\core_perl;C:\Program Files\Java\jre1.8.0_261\bin;C:\msys64\home\david\java\fop

You see ...;C:\msys64\usr\bin;C:\msys64\usr\bin;... whereas I would expect ...;C:\msys64\usr\bin;C:\msys64\bin;...

[codingdave]

Coming back to my Q2, if I remove -E env LANG=C LC_ALL=C bash from CMakeLists.txt it compiles fine. So why can't we just set LC_ALL=C and LANG=C in the bash script again? The script will be called with /usr/sh as indicated by the documention for the MSYS makefiles.

[phrrk]

Alternatively is there any reason why sh cannot be reinstated? As far as I can see it is what it used to be used in CMakeFiles.txt and then was removed earlier this month, I guess to get rid of perceived redundancy (under linux) forgetting that windows required a shell to run the script in. Once this problem was identified, rather than reverse the commit, bash was introduced instead. An alternative would be to have an if(WIN32) ... else block, but I guess that is a bit ugly.

[codingdave]

This is basically the finding of bb388fdc0513c2564352b171dea27311da9b352a. sh should be necessary for MinGW64 If this broke another system we need to understand what why. So @TurboGit can you please elaborate on da5f15562c0c6ada5a49c537e9c6fba7326f0d6f?

[codingdave]

Q2 seems to be answered by 2fac7a457bf7f3b84b6dd9422937cf72ca29fe95 and the background can be seen here: https://github.com/travis-ci/travis-ci/issues/8812#issuecomment-347457115

[parafin]

1. bash was used instead of sh because shebang contains bash and not sh (they may be different)
2. authors_h.sh shebang can be changed to /bin/sh and bash in cmake then to sh (though it's unclear why this helps)
3. LANG and LC_ALL can be set inside the script
4. if we support not only msys on windows, then adding sh or bash is necessary in cmake

If replacing bash with sh works for everyone, then I suggest to go that way and not care about bash doesn't work. So change shebang to /bin/sh in the script and use sh instead of bash in cmake. Optionally move LANG and LC_ALL to script itself (where they belong IMHO) and get rid of cmake -E env. Though I'm not sure if just sh will work without cmake -E env on Windows.

[parafin]

Actually I'm wrong, we can't change bash to sh. I misunderstood how checkbashisms tool works, there are actually bashisms there. So to change bash to sh we need to correct the script too.

[parafin]

What do the following commands print for you (correct the path to authors_h.sh script):

cmake -E env sh -c ls C:/msys64/home/jim/darktable/tools/authors_h.sh
cmake -E env bash -c ls C:/msys64/home/jim/darktable/tools/authors_h.sh
cmake -E env /bin/sh -c ls C:/msys64/home/jim/darktable/tools/authors_h.sh
cmake -E env /bin/bash -c ls C:/msys64/home/jim/darktable/tools/authors_h.sh
cmake -E env /usr/bin/sh -c ls C:/msys64/home/jim/darktable/tools/authors_h.sh
cmake -E env /usr/bin/bash -c ls C:/msys64/home/jim/darktable/tools/authors_h.sh

[phrrk]

@parafin The answer to what you proposed is towards the end, but I suspect that you may have meant what is directly below

jim@Jim-PC MINGW64 ~/darktable/build $ cmake -E env sh -c "ls C:/msys64/home/jim/darktable/tools/authors_h.sh" C:/msys64/home/jim/darktable/tools/authors_h.sh

jim@Jim-PC MINGW64 ~/darktable/build $ cmake -E env bash -c "ls C:/msys64/home/jim/darktable/tools/authors_h.sh" ls: cannot access 'C:/msys64/home/jim/darktable/tools/authors_h.sh': No such file or directory

jim@Jim-PC MINGW64 ~/darktable/build $ cmake -E env /bin/sh -c "ls C:/msys64/home/jim/darktable/tools/authors_h.sh" C:/msys64/home/jim/darktable/tools/authors_h.sh

jim@Jim-PC MINGW64 ~/darktable/build $ cmake -E env /bin/bash -c "ls C:/msys64/home/jim/darktable/tools/authors_h.sh" C:/msys64/home/jim/darktable/tools/authors_h.sh

jim@Jim-PC MINGW64 ~/darktable/build $ cmake -E env /usr/bin/sh -c "ls C:/msys64/home/jim/darktable/tools/authors_h.sh" C:/msys64/home/jim/darktable/tools/authors_h.sh

jim@Jim-PC MINGW64 ~/darktable/build $ cmake -E env /usr/bin/bash -c "ls C:/msys64/home/jim/darktable/tools/authors_h.sh" C:/msys64/home/jim/darktable/tools/authors_h.sh

---

jim@Jim-PC MINGW64 ~/darktable/build $ cmake -E env sh -c ls C:/msys64/home/jim/darktable/tools/authors_h.sh cmake_install.cmake compile_commands.json doc src cmake_uninstall.cmake CPackConfig.cmake Makefile tools CMakeCache.txt CPackSourceConfig.cmake packaging CMakeFiles data po

jim@Jim-PC MINGW64 ~/darktable/build $ cmake -E env bash -c ls C:/msys64/home/jim/darktable/tools/authors_h.sh CMakeCache.txt CMakeFiles CPackConfig.cmake CPackSourceConfig.cmake Makefile cmake_install.cmake cmake_uninstall.cmake compile_commands.json data doc packaging po src tools

jim@Jim-PC MINGW64 ~/darktable/build $ cmake -E env /bin/sh -c ls C:/msys64/home/jim/darktable/tools/authors_h.sh cmake_install.cmake compile_commands.json doc src cmake_uninstall.cmake CPackConfig.cmake Makefile tools CMakeCache.txt CPackSourceConfig.cmake packaging CMakeFiles data po

jim@Jim-PC MINGW64 ~/darktable/build $ cmake -E env /bin/bash -c ls C:/msys64/home/jim/darktable/tools/authors_h.sh cmake_install.cmake compile_commands.json doc src cmake_uninstall.cmake CPackConfig.cmake Makefile tools CMakeCache.txt CPackSourceConfig.cmake packaging CMakeFiles data po

jim@Jim-PC MINGW64 ~/darktable/build $ cmake -E env /usr/bin/sh -c ls C:/msys64/home/jim/darktable/tools/authors_h.sh cmake_install.cmake compile_commands.json doc src cmake_uninstall.cmake CPackConfig.cmake Makefile tools CMakeCache.txt CPackSourceConfig.cmake packaging CMakeFiles data po

jim@Jim-PC MINGW64 ~/darktable/build $ cmake -E env /usr/bin/bash -c ls C:/msys64/home/jim/darktable/tools/authors_h.sh cmake_install.cmake compile_commands.json doc src cmake_uninstall.cmake CPackConfig.cmake Makefile tools CMakeCache.txt CPackSourceConfig.cmake packaging CMakeFiles data po

jim@Jim-PC MINGW64 ~/darktable/build $

[codingdave]

I second @phrrk . Same output on my side.

[parafin]

Right, thanx. So cmake -E env bash is different for some reason. That's very strange... So how about these commands:

cmake -E env bash -c "echo $SHELL $BASH"
cmake -E env which bash
cmake -E env bash --version
cmake -E env /bin/bash --version

[phrrk]

very interesting - cmake finds a different bash!

jim@Jim-PC MINGW64 ~/darktable/build $ cmake -E env bash -c "echo $SHELL $BASH" /usr/bin/bash /usr/bin/bash

jim@Jim-PC MINGW64 ~/darktable/build $ make -E env which bash make: *** missing separator. Stop.

jim@Jim-PC MINGW64 ~/darktable/build $ cmake -E env bash --version GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu) Copyright (C) 2019 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html

This is free software; you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

jim@Jim-PC MINGW64 ~/darktable/build $ cmake -E env /bin/bash --version GNU bash, version 4.4.23(2)-release (x86_64-pc-msys) Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html

This is free software; you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

jim@Jim-PC MINGW64 ~/darktable/build $ /bin/bash --version GNU bash, version 4.4.23(2)-release (x86_64-pc-msys) Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html

This is free software; you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

jim@Jim-PC MINGW64 ~/darktable/build $ /usr/bin/bash --version GNU bash, version 4.4.23(2)-release (x86_64-pc-msys) Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html

This is free software; you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

[parafin]

So it's a different bash, but it still reports itself as /usr/bin/bash. But where else can this bash be? Could you search for bash in the whole Windows filesystem? What about if you try starting bash from cmd.exe?..

[phrrk]

Yep - cmake is finding bash in windows

C:\Users\jim>bash --version GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu) Copyright (C) 2019 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html

This is free software; you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

C:\Users\jim>which bash /c/WINDOWS/system32/bash

C:\Users\jim>

[todd-prior]

@chhil Did you manage to get things going I tried to follow the thread. I didn't alter my build environment but I did delete all my build files and still no luck...during the build I get stuck as I noted you did earlier here....

[parafin]

What about these:

cmake -E env env bash --version
which cmake
cmake --version
cmake -E env cmake --version
cmake -E env env cmake --version

[TurboGit]

If this broke another system we need to understand what why. So @TurboGit can you please elaborate on da5f155?

I did that as the script is in bash and shebang does not work on Windows. I after that added bash for Windows.

[TurboGit]

So why can't we just set LC_ALL=C and LANG=C in the bash script again?

We can do that. Not sure about the again here, this was done long time ago IIRC.

[todd-prior]

@TurboGit Sorry to be an idiot would the output of the commands suggested buy @parafin help or any of the logs??

[todd-prior]

@parafin make -E env env bash --version GNU bash, version 4.4.23(2)-release (x86_64-pc-msys)

[parafin]

One more command I would like to see an output from:

cmake -E environment

[todd-prior]

@parafin I only *** out my onedrive name and subst "user" in place of my username

$ cmake -E environment !;=;\ ACLOCAL_PATH=C:\apps\msys64\mingw64\share\aclocal;C:\apps\msys64\usr\share\aclocal ALLUSERSPROFILE=C:\ProgramData APPDATA=C:\Users\user\AppData\Roaming CAMLIBS=C:/apps/msys64/mingw64/lib/libgphoto2/2.5.23/ CLASSPATH=;C:\apps\msys64\home\user\java\saxon\saxon9he.jar;C:\apps\msys64\home\user\java\docbook-xsl-saxon\saxon65.jar COMMONPROGRAMFILES=C:\Program Files\Common Files COMPUTERNAME=DESKTOP-GQDS5F4 COMSPEC=C:\WINDOWS\system32\cmd.exe CONFIG_SITE=C:/apps/msys64/mingw64/etc/config.site CONTITLE=MinGW x64 CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files CommonProgramW6432=C:\Program Files\Common Files DriverData=C:\Windows\System32\Drivers\DriverData FPS_BROWSER_APP_PROFILE_STRING=Internet Explorer FPS_BROWSER_USER_PROFILE_STRING=Default HOME=C:\apps\msys64\home\user HOMEDRIVE=C: HOMEPATH=\Users\user HOSTNAME=DESKTOP-GQDS5F4 ICONV=C:\DevLibs\iconv\bin INFOPATH=C:\apps\msys64\usr\local\info;C:\apps\msys64\usr\share\info;C:\apps\msys64\usr\info;C:\apps\msys64\share\info IOLIBS=C:/apps/msys64/mingw64/lib/libgphoto2_port/0.12.0/ LANG=en_US.UTF-8 LD_LIBRARY_PATH=C:\apps\msys64\mingw64\lib;. LOCALAPPDATA=C:\Users\user\AppData\Local LOGINSHELL=bash LOGONSERVER=\DESKTOP-GQDS5F4 MAGICK_HOME=C:\Program Files\DT_Built\lib\GraphicsMagick-1.3.35\modules-Q8\coders MANPATH=C:\apps\msys64\mingw64\local\man;C:\apps\msys64\mingw64\share\man;C:\apps\msys64\usr\local\man;C:\apps\msys64\usr\share\man;C:\apps\msys64\usr\man;C:\apps\msys64\share\man MINGW_CHOST=x86_64-w64-mingw32 MINGW_PACKAGE_PREFIX=mingw-w64-x86_64 MINGW_PREFIX=C:/apps/msys64/mingw64 MSYSCON=mintty.exe MSYSTEM=MINGW64 MSYSTEM_CARCH=x86_64 MSYSTEM_CHOST=x86_64-w64-mingw32 MSYSTEM_PREFIX=C:/apps/msys64/mingw64 NUMBER_OF_PROCESSORS=8 OLDPWD=C:/apps/msys64/home/user/darktable ORIGINAL_PATH=C:\Windows\System32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\ ORIGINAL_TEMP=C:/Users/user/AppData/Local/Temp ORIGINAL_TMP=C:/Users/user/AppData/Local/Temp OS=Windows_NT OneDrive=* OneDriveCommercial=**** OneDriveConsumer=***** PATH=C:\apps\msys64\mingw64\bin;C:\apps\msys64\usr\local\bin;C:\apps\msys64\usr\bin;C:\apps\msys64\usr\bin;C:\Windows\System32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\apps\msys64\usr\bin\site_perl;C:\apps\msys64\usr\bin\vendor_perl;C:\apps\msys64\usr\bin\core_perl;C:\Program Files\Java\jre1.8.0_261\bin;C:\apps\msys64\home\user\java\fop PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC PKG_CONFIG_PATH=C:\apps\msys64\mingw64\lib\pkgconfig;C:\apps\msys64\mingw64\share\pkgconfig PREFIX=C:/apps/msys64/mingw64 PRINTER=Microsoft Print to PDF PROCESSOR_ARCHITECTURE=AMD64 PROCESSOR_IDENTIFIER=AMD64 Family 21 Model 2 Stepping 0, AuthenticAMD PROCESSOR_LEVEL=21 PROCESSOR_REVISION=0200 PROGRAMFILES=C:\Program Files PROMPT=$P$G PS1=[\e]0;\w\a]\n[\e[32m]\u@\h [\e[35m]$MSYSTEM[\e[0m] [\e[33m]\w[\e[0m]\n\$ PSModulePath=C:\Program Files\WindowsPowerShell\Modules;C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules PUBLIC=C:\Users\Public PWD=C:/apps/msys64/home/user/darktable/build ProgramData=C:\ProgramData ProgramFiles(x86)=C:\Program Files (x86) ProgramW6432=C:\Program Files SAXON_INSTALL_DIR=C:/apps/msys64/home/user SESSIONNAME=Console SHELL=C:/apps/msys64/usr/bin/bash SHLVL=1 SYSTEMDRIVE=C: SYSTEMROOT=C:\WINDOWS TEMP=C:\apps\msys64\tmp TERM=xterm TERM_PROGRAM=mintty TERM_PROGRAM_VERSION=3.3.0 TMP=C:\apps\msys64\tmp USER=user USERDOMAIN=DESKTOP-GQDS5F4 USERDOMAIN_ROAMINGPROFILE=DESKTOP-GQDS5F4 USERNAME=user USERPROFILE=C:\Users\user VBOX_MSI_INSTALLPATH=C:\Program Files\Oracle\VirtualBox\ WD=C:\apps\msys64\usr\bin\ WINDIR=C:\WINDOWS =C:/apps/msys64/mingw64/bin/cmake dcamprof=C:\Program FIles\dcamprof\ libxml2=C:\DevLibs\libxml2\bin temp=C:\Users\user\AppData\Local\Temp tmp=C:\Users\user\AppData\Local\Temp zlib=C:\DevLibs\zlib\bin

[todd-prior]

CMakeError.log CMakeOutput.log

[parafin]

OK, so this is what happens in my understanding:

1. cmake uses CreateProcessW to start programs
2. this API call looks in Windows system directory before consulting the $PATH variable
3. another version of bash is located there (where did it come from?), compiled differently and thus not understanding the path with drive in it
4. using full path to bash or sh, which doesn't exist in Windows system directory, avoids this issue
5. so we need a way to make sure that msys installation is preferred over Windows system directory, basically we want $PATH to be a priority
6. I have no idea how mingw/msys is implemented, but it seems starting bash through env tool (not to be confused with cmake -E env feature) finds the correct executable (that is until some other env is put in Windows system directory)
7. another alternative is to experiment with find_program cmake call to get full path to bash - it might be more correct way to solve it
8. ....
9. profit

[todd-prior]

Nice detective work ….beyond my skill set….

From: parafin notifications@github.com Sent: October 15, 2020 3:44 PM To: darktable-org/darktable darktable@noreply.github.com Cc: Prior,Todd priort@mcmaster.ca; Comment comment@noreply.github.com Subject: Re: [darktable-org/darktable] Windows Build Issue (#6487)

OK, so this is what happens in my understanding:

1. cmake uses CreateProcessWhttps://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessw to start programs
2. this API call looks in Windows system directory before consulting the $PATH variable
3. another version of bash is located there (where did it come from?), compiled differently and thus not understanding the path with drive in it
4. using full path to bash or sh, which doesn't exist in Windows system directory, avoids this issue
5. so we need a way to make sure that msys installation is preferred over Windows system directory, basically we want $PATH to be a priority
6. I have no idea how mingw/msys is implemented, but it seems starting bash through env tool (not to be confused with cmake -E env feature) finds the correct executable (that is until some other env is put in Windows system directory)
7. another alternative is to experiment with find_program cmake call to get full path to bash - it might be more correct way to solve it
8. ....
9. profit

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/darktable-org/darktable/issues/6487#issuecomment-709550490, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AKS2ES5R2RBNURHD3AH6SOLSK5GGZANCNFSM4SKAOUVQ.

[codingdave]

cmake -E env env bash --version which cmake cmake --version cmake -E env cmake --version cmake -E env env cmake --version

$ cmake -E env env bash --version
GNU bash, version 4.4.23(2)-release (x86_64-pc-msys)

$ which cmake /mingw64/bin/cmake

$ cmake --version cmake version 3.17.3

$ cmake -E env cmake --version cmake version 3.17.3

$ cmake -E env env cmake --version cmake version 3.17.3

> One more command I would like to see an output from:
> 
> ```
> cmake -E environment
> ```

$ cmake -E environment !;=;\ ALLUSERSPROFILE=C:\ProgramData ANDROID_SDK_HOME=C:\Android APPDATA=C:\Users\david\AppData\Roaming CAMLIBS=C:/msys64/mingw64/lib/libgphoto2/2.5.23/ CLASSPATH=C:\msys64\java\saxon\saxon10he.jar;C:\msys64\home\david\java\docbook-xsl-saxon\saxon65.jar CMDER_ROOT=E:\storage\Windows-Base-programs\cmder COMMONPROGRAMFILES=C:\Program Files\Common Files COMPUTERNAME=FOOBAR COMSPEC=C:\WINDOWS\system32\cmd.exe CONFIG_SITE=C:/msys64/etc/config.site CONTITLE=MSYS2 MSYS ChocolateyInstall=C:\ProgramData\chocolatey ChocolateyLastPathUpdate=132206329538457761 ChocolateyPath=C:\Chocolatey CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files CommonProgramW6432=C:\Program Files\Common Files ConEmuDir=E:\storage\Windows-Base-programs\cmder\vendor\conemu-maximus5 DriverData=C:\Windows\System32\Drivers\DriverData FPS_BROWSER_APP_PROFILE_STRING=Internet Explorer FPS_BROWSER_USER_PROFILE_STRING=Default GIT_LFS_PATH=C:\Program Files\Git LFS GIT_SSH=C:\Program Files (x86)\PuTTY\plink.exe GTK_BASEPATH=C:\Program Files (x86)\GtkSharp\2.12\ HOME=C:\msys64\home\david HOMEDRIVE=C: HOMEPATH=\Users\david HOSTNAME=foobar INFOPATH=C:\msys64\usr\local\info;C:\msys64\usr\share\info;C:\msys64\usr\info;C:\msys64\share\info IOLIBS=C:/msys64/mingw64/lib/libgphoto2_port/0.12.0/ JAVA_HOME=C:/Program Files/Java/jre1.8.0_261 LANG=en_US.UTF-8 LD_LIBRARY_PATH=C:\msys64\mingw64\lib;. LOCALAPPDATA=C:\Users\david\AppData\Local LOGINSHELL=bash LOGONSERVER=\FOOBAR MANPATH=C:\msys64\usr\local\man;C:\msys64\usr\share\man;C:\msys64\usr\man;C:\msys64\share\man MSYSCON=mintty.exe MSYSTEM=MSYS MSYSTEM_CARCH=x86_64 MSYSTEM_CHOST=x86_64-pc-msys MSYSTEM_PREFIX=C:/msys64/usr NUMBER_OF_PROCESSORS=12 OLDPWD=C:/msys64/home/david/darktable ORIGINAL_PATH=C:\Windows\System32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\ ORIGINAL_TEMP=C:/Users/david/AppData/Local/Temp ORIGINAL_TMP=C:/Users/david/AppData/Local/Temp OS=Windows_NT OneDrive=FOOO OneDriveCommercial=FOOO PATH=C:\msys64\mingw64\bin;C:\msys64\usr\local\bin;C:\msys64\usr\bin;C:\msys64\usr\bin;C:\msys64\opt\bin;C:\Windows\System32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\msys64\usr\bin\site_perl;C:\msys64\usr\bin\vendor_perl;C:\msys64\usr\bin\core_perl;C:\Program Files\Java\jre1.8.0_261\bin;C:\msys64\home\david\java\fop PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.py;.pyw PKG_CONFIG_PATH=C:\msys64\usr\lib\pkgconfig;C:\msys64\usr\share\pkgconfig;C:\msys64\lib\pkgconfig POWERSHELL_DISTRIBUTION_CHANNEL=MSI:Windows 10 Pro PREFIX=C:/msys64/mingw64 PRINTER=FOOOO PROCESSOR_ARCHITECTURE=AMD64 PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 79 Stepping 1, GenuineIntel PROCESSOR_LEVEL=6 PROCESSOR_REVISION=4f01 PROGRAMFILES=C:\Program Files PROMPT=$P$G PS1=[\e]0;\w\a]\n[\e[32m]\u@\h [\e[35m]$MSYSTEM[\e[0m] [\e[33m]\w[\e[0m]\n\$ PSModulePath=C:\Program Files\WindowsPowerShell\Modules;C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules;C:\Program Files (x86)\Microsoft SQL Server\140\Tools\PowerShell\Modules\ PUBLIC=C:\Users\Public PWD=C:/msys64/home/david/darktable/build PreferredToolArchitecture=x64 ProgramData=C:\ProgramData ProgramFiles(x86)=C:\Program Files (x86) ProgramW6432=C:\Program Files SAXON_INSTALL_DIR=C:/msys64/home/david SESSIONNAME=Console SHELL=C:/msys64/usr/bin/bash SHLVL=1 SONAR_HOME=D:\CI\sonarqube SVN_SSH=C:\Program Files (x86)\PuTTY\plink.exe SYSTEMDRIVE=C: SYSTEMROOT=C:\WINDOWS TEMP=C:\msys64\tmp TERM=xterm TERM_PROGRAM=mintty TERM_PROGRAM_VERSION=3.4.0 TMP=C:\msys64\tmp UGII_BASE_DIR=C:\Program Files\Siemens\NX 11.0 UGII_LANG=english USER=david USERDOMAIN=FOOO USERDOMAIN_ROAMINGPROFILE=FOOO USERNAME=david USERPROFILE=C:\Users\david VBOX_MSI_INSTALLPATH=C:\Program Files\Oracle\VirtualBox\ VS100COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\Tools\ VS110COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\Tools\ VS120COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\Tools\ VS140COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools\ VS150COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE VS160COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\Common7\Tools VSSDK100Install=C:\Program Files (x86)\Microsoft Visual Studio 2010 SDK SP1\ WD=C:\msys64\usr\bin\ WINDIR=C:\WINDOWS =C:/msys64/mingw64/bin/cmake _IsNativeEnvironment=true temp=C:\Users\david\AppData\Local\Temp tmp=C:\Users\david\AppData\Local\Temp

[parafin]

Ah, just thought of another possibility (not sure if it will work) - remove cmake -E env from add_custom_command and keep simple bash there (we will need to move LANG and LC_ALL into the script itself, or use -c option). If it works, it would be the simplest solution.

[todd-prior]

I have the windows linux kernel enabled…ie wsl2 I wonder if that is the source. I have ubuntu 20.04 installed on wsl2…Maybe that is where it comes from???

From: parafin notifications@github.com Sent: October 15, 2020 3:44 PM To: darktable-org/darktable darktable@noreply.github.com Cc: Prior,Todd priort@mcmaster.ca; Comment comment@noreply.github.com Subject: Re: [darktable-org/darktable] Windows Build Issue (#6487)

OK, so this is what happens in my understanding:

1. cmake uses CreateProcessWhttps://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessw to start programs
2. this API call looks in Windows system directory before consulting the $PATH variable
3. another version of bash is located there (where did it come from?), compiled differently and thus not understanding the path with drive in it
4. using full path to bash or sh, which doesn't exist in Windows system directory, avoids this issue
5. so we need a way to make sure that msys installation is preferred over Windows system directory, basically we want $PATH to be a priority
6. I have no idea how mingw/msys is implemented, but it seems starting bash through env tool (not to be confused with cmake -E env feature) finds the correct executable (that is until some other env is put in Windows system directory)
7. another alternative is to experiment with find_program cmake call to get full path to bash - it might be more correct way to solve it
8. ....
9. profit

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/darktable-org/darktable/issues/6487#issuecomment-709550490, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AKS2ES5R2RBNURHD3AH6SOLSK5GGZANCNFSM4SKAOUVQ.