gganssauge
commented
4 years ago Once again: zip file mentioned in the issue: scons-bug.zip
Closed gganssauge closed 7 months ago
gganssauge
commented
4 years ago Once again: zip file mentioned in the issue: scons-bug.zip
mwichmann
commented
7 months ago Working on a variety of encoding issues and came across this one again. Does this change sound reasonable? It reads the temporary files as bytes, and tries to decode into whatever the codec for the two output streams are, defining an error behavior to avoid bailing on something that can't convert ("backslashreplace" is another option).
diff --git a/SCons/Platform/win32.py b/SCons/Platform/win32.py
index b14582361..c2216eb68 100644
--- a/SCons/Platform/win32.py
+++ b/SCons/Platform/win32.py
@@ -165,16 +165,18 @@ def piped_spawn(sh, escape, cmd, args, env, stdout, stderr):
# and do clean up stuff
if stdout is not None and not stdoutRedirected:
try:
- with open(tmpFileStdoutName) as tmpFileStdout:
- stdout.write(tmpFileStdout.read())
+ with open(tmpFileStdoutName, "rb") as tmpFileStdout:
+ output = tmpFileStdout.read()
+ stdout.write(output.decode(sys.stdout.encoding, "replace"))
os.remove(tmpFileStdoutName)
except OSError:
pass
if stderr is not None and not stderrRedirected:
try:
- with open(tmpFileStderrName) as tmpFileStderr:
- stderr.write(tmpFileStderr.read())
+ with open(tmpFileStderrName, "rb") as tmpFileStderr:
+ errors = tmpFileStderr.read()
+ stdout.write(errors.decode(sys.stderr.encoding, "replace"))
os.remove(tmpFileStderrName)
except OSError:
pass
bdbaddog
commented
7 months ago @mwichmann looks good. does it work with the tarball they attached?
mwichmann
commented
7 months ago I don't have a German Windows, so hard to repro. I'll take a look. Manually following the steps (the objectionable message stored in a file) looks like this. Notice since my Windows is set to a US codepage, it doesn't fail on the message, but it displays it wonky; the decode inside Python made it come out okay.
C:\Users\mats\Documents\github\scons\exp\encoded-response>type data
.sconf_temp\conftest_0.c(11): warning C4163: "filelength": Nicht als systeminterne Funktion verf├╝gbar
C:\Users\mats\Documents\github\scons\exp\encoded-response>py
Python 3.12.1 (tags/v3.12.1:2305ca5, Dec 7 2023, 22:03:25) [MSC v.1937 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> with open("data", "rb") as f:
... data = f.read()
...
>>>
>>> sys.stdout.write(data.decode(sys.stdout.encoding, errors="convert"))
.sconf_temp\conftest_0.c(11): warning C4163: "filelength": Nicht als systeminterne Funktion verfügbar
102
>>>
Encoding issue on german windows 10 when running configure
I discussed the issue with @bdbaddog on the #scons-help channel on discord and he asked me to add a bug report here.
Table of Contents
Problem statement
Consider the following SConstruct
when this is run the
filelengthfunction is not detected, although it is clearly part of the Visual-C++ runtime library.Environment
I tried two different editions of Visual Studio: Visual Studio Express 2017 and Visual Studio 2019
Analysis
When looking at the generated config.log file the reason becomes apparent:
Workaround
As a workaround I applied the following patch:
Conclusion
Another run with the workaround applied yields the following:
The character ü above is actually the character with code 0x81 mentioned in the stacktrace of the first run.
filelengthnot being a function doesn't make it unusable, as the now working check proves.Reproducing the bug
In order to reproduce the bug you need a computer (or a VM) running a localized version of Windows 10 with Visual Studio and Python 3 installed.
The attached zip archive contains a windows batch script
bug.cmd, theSConstructfile, a patched version ofSCons\Platform\win32.pyand this bug report.As a convenience I added the output of the warning to the file
warning.txtin original cp850 encoding.After unpacking the archive into an empty directory, simply run
bug.cmd. It creates a virtualenv with the latest scons installed, runs scons to demonstrate the problem, copies a patched version ofSCons.Platform.win32to the scons directory and runs scons again in order to demonstrate that now the bug is gone.