search

python / cpython

The Python programming language
https://www.python.org
Other
63.12k stars 30.22k forks source link

Building python 3.10.12, 3.11.4, 3.12.0b2 on Windows with a case sensitive file system fails #105737

Open tps800 opened 1 year ago

tps800 commented 1 year ago

Bug report

Trying to compile python 3.10.12 (as trying to compile 3.11.4, 3.12.0b2) on windows leads to following error:

  _testimportmultiple.vcxproj -> C:\Users\user\Documents\Projekte\cpython\PCbuild\amd64\_testimportmultiple.pyd
  _testmultiphase.c
     Bibliothek "C:\Users\user\Documents\Projekte\cpython\PCbuild\amd64\_testmultiphase.lib" und Objekt "C:\Users\user\Documents\Projekte\cpython\PCbuild\amd64\_testmultiphase.exp" werden erstellt.
  Code wird generiert.
  Codegenerierung ist abgeschlossen.
  _testmultiphase.vcxproj -> C:\Users\user\Documents\Projekte\cpython\PCbuild\amd64\_testmultiphase.pyd
  _testconsole.c
C:\Users\user\Documents\Projekte\cpython\PC\_testconsole.c(13,10): fatal  error C1083: Datei (Include) kann nicht geöffnet werden: "..\modules\_io\_iomodule.h": No such file or directory [C:\Users\user\Documents\Projekte\cpython\PCbuild\_testconsole.vcxproj]

Fehler beim Buildvorgang.

Renaming Modules to modules makes the error vanish, but leads to directory ..\Modules\... not found at other places.

Your environment

eryksun commented 1 year ago

In 3.11, the include in "PC/_testconsole.c" should target "..\Modules\_io\_iomodule.h" instead of "..\modules\_io\_iomodule.h".

https://github.com/python/cpython/blob/v3.11.4/PC/_testconsole.c#L13

This isn't a problem in 3.12+ since "PC/_testconsole.c" no longer includes "Modules/_io/_iomodule.h".

zooba commented 1 year ago

Contributions welcome. This is outside of our normal supported dev environment.

If it doesn't apply to 3.13, then start with a 3.12 PR and we can backport from there.

aisk commented 5 months ago

TL;DR: In Python 3.12+, Python can be built with a case-sensitive filesystem, but the incremental build appears to be broken.

I created a new folder, set it to be case-sensitive, and updated the Git config as described at https://learn.microsoft.com/en-us/windows/wsl/case-sensitivity.

Python main branch (978fba58aef347de4a1376e525df2dacc7b2fff3) can be build on this folder, except these warnings:

C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(254,5): warning MSB8064: 项“C:\Users\xxxxx\case-sensitive\cpython\externals\\mpdecimal-4.0.0\\libmpdec\vcdiv64.asm”的自定义生成成功，但指定的依赖项“c:\users\xxxxx\case-sensitive\cpython\externals\mpdecimal-4.0.0\libmpdec\vcdiv64.asm”不存在。这可能会导致增量生成无法正常工作。 [C:\Users\xxxxx\case-sensitive\cpython\PCbuild\_decimal.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(254,5): warning MSB8065: 项“C:\Users\xxxxx\case-sensitive\cpython\externals\\mpdecimal-4.0.0\\libmpdec\vcdiv64.asm”的自定义生成成功，但尚未创建指定的输出“c:\users\xxxxx\case-sensitive\cpython\pcbuild\obj\313amd64_debug\_decimal\vcdiv64.obj”。这可能会导致增量生成无法正常工作。 [C:\Users\xxxxx\case-sensitive\cpython\PCbuild\_decimal.vcxproj]

The localized Windows messages state that some targets were built successfully, but the output file does not exist, and the incremental generation may woult work. I suspect this issue is due to the path name discrepancy (pcbuild instead of PCbuild, users instead of Users). This also results in PCbuild\build.bat recompiling almost the entire project.

Therefore, in Python 3.12+, Python can be built with a case-sensitive filesystem, but it seems that the incremental build process is malfunctioning.

zooba commented 5 months ago

It looks like something is calculating an output filename, normalising it (multiple backslashes are also combined) and lowercasing it (the case differences noted).

I'm not sure that we're doing that in any of our headers? It might be a VS issue.

aisk commented 5 months ago

I'm not familiar with MSBuild, but I found that the target is caculated from CustomBuild's include, which always returns lowercased target name: https://github.com/python/cpython/blob/c4f9823be277b2e3de2315526612276626217743/PCbuild/_decimal.vcxproj#L144-L150

Current MSBuilld version is 17.9.8+b34f75857 on my machine. Maybe we can wait to see Microsoft fix this.

zooba commented 5 months ago

I wouldn't bet on a fix anytime soon, at least not until something much more important than us needs to work on a case-sensitive file system.

More likely we need to create a different Target to run the ml64 command and assign the output metadata ourselves, to bypass whatever extra step is happening in the C++ targets files.