[vcpkg-tool] build failure for some ports (libiconv, gettext) I have found so far

[StarGate-One]

Host Environment

• OS: Windows Version [10.0.19044.1566] Pro x64
• Compiler: revision
  • Visual Studio Community 2022 Version: [17.1.0] VS Tools Version: [14.31.31103]
  • Microsoft (R) C/C++ Optimizing Compiler Version [19.31.31104] for x64
  • Other Software versions:
  • CMake Version [3.22.2]
  • Git Version [2.35.1.windows.2]
  • Ninja Version [1.10.2]
  • Visual Studio Locator (vswhere) Version [2.9.3+21bcdb639c] query version [3.1.1126.51066]
  • Windows SDK Version [10.0.22000.0]

To Reproduce Steps to reproduce the behavior: ./vcpkg install libiconv:x64-windows ./vcpkg install gettext:x64-windows

• Vcpkg Version [2022-02-24-7efe7f2b04912a7d674f86a406f9f2711c0221c6] - ports libconv and gettext build successfully
• Vcpkg Version [2022-03-03-6ef3648a2b7daf197fce82d936ff88ef98c26361] - ports libiconv and gettext fail with the following in the config-x64-windows-dbg-err.log file: ./../src/xxxx.clean/configure: line 478: sed: command not found ./../src/xxxx.clean/configure: line 477: expr: command not found ./../src/xxxx.clean/configure: line 478: sed: command not found ./../src/xxxx.clean/configure: line 492: sed: command not found : error: cannot create .lineno; rerun with a POSIX shell

Failure logs -Cut and paste the appropriate build messages from the console output - in zip file -Please attach any additional failure logs mentioned in the console output - in zip file Included port libiconv console log along with the complete .\buildtrees\libiconv folder minus the src libiconv.zip

Additional context It seems a change made in between these 2 release versions of the vcpkg-tool something has caused the bash command to fail somehow? The bootstrap-vcpkg.bat is still downloading the 2022-02-24 version of vcpkg-tool, but I was testing the 2022-03-03 version.

[dg0yt]

Did you already inspect or remove D:/vcpkg-test/downloads/tools/msys2/7e05e7aa09f1709f? Can you share the config.log file either from the x64-windows-dbg folder or as config.log... from buildtrees/libiconv?

[StarGate-One]

Did you already inspect or remove D:/vcpkg-test/downloads/tools/msys2/7e05e7aa09f1709f? fresh/clean clone of vcpkg for each version of vcpkg - both download and install the pre-requisite msys2 tools in the /downloads/tools/msys2 directory Can you share the config.log file either from the x64-windows-dbg folder or as config.log... from buildtrees/libiconv? included libiconv.zip file in the original post, above, but most of the files are 0 byte size..

• I took 2 vanilla clones of vcpkg, download the 2022-03-03 vcpkg-tool in one 2022-02-24 vcpkg-tool in the vcpkg root - libconv and gettext fail to install in the clone containing 2022-03-03 vcpkg-tool and successfully install in the clone containing 2022-02-24 vcpkg-tool.

• I then vcpkg remove the gettext and libiconv in the clone with the 2022-02-24 vcpkg-tool and replaced it with the 2022-03-03 vcpkg-tool and again the clone with 2022-02-24 successfully installed the 2 ports where the one now with the 2022-03-03 now fails install on the 2 ports.

• If you wish not to download/unzip and inspect its content, you can always clone a vcpkg to another repo, download the 2022-03-03 vcpkg-tool and put in the vcpkg repo root and try to install libconv and gettext and see if you obtain the same results?

• I discovered this issue while testing the vcpkg-tool with the latest commits, as I compile my own version of the vcpkg-tool with local customizations, so when these two ports failed, I first compiled the vcpkg-tool without my customizations and netted the same results... So I download the 2022-03-03 vcpkg-tool release and it again failed with these same 2 ports. I dropped down to the 2022-02-24 vcpkg-tool release and the 2 ports installed successfully.

• The end result is for whatever reason the bash shell command(s) seems to be failing only on these 2 ports, like it is trying to execute the sed and expr from outside the bash shell itself.

• I have run across, on these 2 port, but I only use a very small number of ports from vcpkg ports/features, so there could be others.

• Again this is within the above zip file, in the file stdout-x64-windows.log:

  -- Configuring x64-windows-dbg CMake Error at scripts/cmake/vcpkg_execute_required_process.cmake:128 (message): Command failed: D:/vcpkg-test/downloads/tools/msys2/7e05e7aa09f1709f/usr/bin/bash.exe --noprofile --norc --debug -c > "V=1 CPP='compile cl.exe -E' CC='compile cl.exe' CC_FOR_BUILD='compile cl.exe' CXX='compile cl.exe' RC='windres-rc rc.exe' WINDRES='windres-rc rc.exe' AR='ar-lib lib.exe' LD='link.exe -verbose' RANLIB=':' STRIP=':' NM='dumpbin.exe -symbols -headers' DLLTOOL='link.exe -verbose -dll' CCAS=':' AS=':' ./../src/1.16-b0c41d3469.clean/configure --build=x86_64-pc-mingw32 \"--enable-extra-encodings\" \"--without-libiconv-prefix\" \"--without-libintl-prefix\" \"--enable-relocatable\" \"ac_cv_prog_ac_ct_STRIP=:\" \"gl_cv_double_slash_root=yes\" \"ac_cv_func_memmove=yes\" \"--disable-silent-rules\" \"--verbose\" \"--enable-shared\" \"--disable-static\" \"--prefix=/D/vcpkg-test/installed/x64-windows/debug\" \"--bindir=\${prefix}/../tools/libiconv/debug/bin\" \"--sbindir=\${prefix}/../tools/libiconv/debug/sbin\" \"--libdir=\${prefix}/lib\" \"--includedir=\${prefix}/../include\" \"--datarootdir=\${prefix}/share/libiconv\"" Working Directory: D:/vcpkg-test/buildtrees/libiconv/x64-windows-dbg Error code: 1 See logs for more information: D:\vcpkg-test\buildtrees\libiconv\config-x64-windows-dbg-err.log

  Call Stack (most recent call first): scripts/cmake/vcpkg_configure_make.cmake:815 (vcpkg_execute_required_process) ports/libiconv/portfile.cmake:29 (vcpkg_configure_make) scripts/ports.cmake:145 (include)

• Thank you for your time and have a wonderful day. /cc @ras0219-msft @BillyONeal @strega-nil-ms

[dg0yt]

• If you wish not to download/unzip and inspect its content, ...

I did that, but config.log* isn't there.

...you can always clone a vcpkg to another repo, download the 2022-03-03 vcpkg-tool and put in the vcpkg repo root and try to install libconv and gettext and see if you obtain the same results?

No, I can't because I don't have Visual Studio. And the ports do build in vcpkg CI, so even if I did, I might not be able reproduce the issue.

Your issue is with tools from MSYS2. MSYS2 is sensitive to mixing different versions of the runtime. So I'm not curiuous about the vpkg tool versions but about other possible sources for these tools. And config.log has details such as the PATH seen inside of the configure script.

[StarGate-One]

• If the log is not there then it is failing before it is getting created...
• The clones are fresh vanilla vcpkg, the only differences is the version of vcpkg-tool
• If you do not have the tools to work this issue, then why are you even involved?

[dg0yt]

• If you do not have the tools to work this issue, then why are you even involved?

I'm a friendly contributor with msys2 and gettext;x64-windows experience. I'm sorry if you feel my help is inappropriate.

[StarGate-One]

• It appears the CI is still running the 2022-02-24 vcpkg-tool version, so of course gettext and libiconv would not fail as the 2022-03-03 version is where the issue surfaces.
• I guess I will just wait and see what happens when the .\bootstrap-vcpkg.bat gets updated to pull the 2022-03-03 version instead of the current 2022-02-24 version.
• I used to be a friendly contributor also, but that is beside the point.
• My reasoning is the issue lies somewhere either with VS2022 vcpkg-tool from 2022-03-03, not with the ports libiconv/gettext themselves.

[StarGate-One]

Ok, did this the hard way, started backing out commits until the vcpkg install libiconv did not fail...

1. $ git reset --hard 6ef3648a2b7daf197fce82d936ff88ef98c26361 HEAD is now at 6ef3648a Use System32\tar.exe if present instead of 7zip for zip archives. (#406) Ports libiconv and gettext fail (gettext will fail due to cascading dependencies of libiconv)
2. $ git reset --hard ca46ecb19fcf9dc45f4c87345a218ce6b264ca76 HEAD is now at ca46ecb1 [localization][automated][ci skip] update locale files (https://github.com/microsoft/vcpkg-tool/commit/ca46ecb19fcf9dc45f4c87345a218ce6b264ca76) Ports libiconv and gettext install successfully.

@dg0yt - Note on the failing libiconv, the .\buildtrees\libiconv\config-x64-windows-dbg-out.log is created as a 0 byte file, the .\buildtrees\libiconv\config-x64-windows-dbg-err.log is created with the sed and expr programs not found and to execute in a POSIX shell. The .\buildtrees\libiconv\x64-windows-dbg directory contains only a single 0 byte file name .lineno. The .\buildtrees\libiconv\x64-windows-dbg\config.h and config.status are not created.

@ras0219-msft @BillyONeal @strega-nil-ms - I studied the changes made in commit Use System32\tar.exe if present instead of 7zip for zip archives. (#406) but with my C/C++ not being that well, I could not identify the exact line(s) causing the failure.

Note these 2 ports use the vcpkg_configure_make and vcpkg_install_make cmake scripts, so I would theorize this may affect some other ports using these same script may also fail with the 2022-03-03 vcpkg-tool version.

I apologize for closing accidently earlier, the purple "Close with Comment" just look more visually appealing the just the green "Comment" button.

[StarGate-One]

Hope this helps, - I think I figured it out...

1. I have a fresh install of Windows 10 Pro 19044:1566, only a few weeks old.
2. It seems somehow is dependent on what shell is looking at the file system at any given time, realizing windows treats C:\Windows\System32 and C:\Windows\system32 the same, when most, if not all nix flavors differentiate between upper and lower case: a. When I do a dir from Windows cmd prompt, system32 is C:\Windows\system32 b. When I look at my environment variables, one of my path values is C:\Windows\system32 c. Comparing the output when I did the .\vcpkg install libiconv --debug >.\somelogname.log 2>&1 when using the 2 different vcpkg versions there are the normal differences, time, process id, etc but the one that caught my attention was the PATH information output was different between the two, more specifically the 2022-02-24 version has C:\Windows\system32 and the 2022-03-03 version has C:\Windows\System32 - see the difference, the one working has lower case system32, the one broke, and upper case System32.
3. So I modified vcpkg-tool src/vcpkg/base/system.cpp and src/vcpkg-test/files.cpp making any occurrence of System32 be system32, recompiled and run my test again - which it then successfully installed libiconv with the latest vcpkg-tool source as of 03:42 UTC 03-07-2022.
4. Now what I do not know is if it breaks anything else???
5. I do have Git installed so I opened a git bash shell and cd /c/Windows/System32 and cd /c/Windows/system32 both work, but when I do cd /c/Windows and do an ls -ald System* I do get all the Windows/System directories, but when I do a ls -ald system* I only get the file system.ini - no System directories. I am not familiar with MSYS2 bash (although git uses it also from my understanding) but git bash may had some additional scripts that allows it to find the Windows System versus Windows system directories.
6. I do not know if this is something that is actually wrong with vcpkg-tool in how it handles getting the system32 files or it finds the Windows tar first while in a bash shell or something amiss in my new windows install?
7. I have attached by test logs, 2022-02-24, 2022-03-03 (broken) and 2022-03-03 (my fixes), screen shot of my compares and the cpp files I changed to make all System32 be system32. vcpkg-2022-02-24.log vcpkg-2022-03-03.log files.cpp.txt system.cpp.txt

[JackBoosY]

@StarGate-One Thanks for the very detailed information. I think case is the reason for this issue. In Windows, the system is not case-sensitive. However in mingW this behavior may be broken. This will cause all port builds that use vcpkg_configure_make in Windows to fail.

Can you please make a PR to https://github.com/microsoft/vcpkg-tool/pulls to fix this?

Thanks!

[StarGate-One]

Could not be repro in CI - closing.

[JackBoosY]

Should wait for next release.

[StarGate-One]

This is fixed now, I close.