build failure on latest 3.10 tag

riccardodivirgilio commented 2 years ago

Describe the bug Unable to finish the build process successfully by running make macOS on a very recent 3.10 tag. https://github.com/beeware/Python-Apple-support/tree/3.10

To Reproduce run make macOS

Environment: Mac Mini M1, macOS 12.0.1, xcode-select version 2392.

Stdout

...
xcrun --sdk macosx clang --sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -mmacosx-version-min=10.15 -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64    -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include    -DPy_BUILD_CORE_BUILTIN  -I./../Support/OpenSSL.xcframework/macos-arm64_x86_64/Headers -DUSE_SSL -c ./Modules/_hashopenssl.c -o Modules/_hashopenssl.o
./Modules/_hashopenssl.c:33:10: fatal error: 'openssl/evp.h' file not found
#include <openssl/evp.h>
         ^~~~~~~~~~~~~~~
1 error generated.
make[1]: *** [Modules/_hashopenssl.o] Error 1
>>> Install Python for macOS
cd build/macOS/Python-3.10.4-macOS && PATH="/Users/testmngr/Downloads/Python-Apple-support-3.10//_install/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/Users/testmngr/Downloads/sqlplus/instantclient_18_1:/usr/local/ruby/rbenv/bin:/Test/apache/apache-ant-1.9.7/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/local/bin" make install 2>&1 | tee -a ../python-macOS.install.log
Makefile:2177: warning: overriding commands for target `Modules/_math.o'
Makefile:624: warning: ignoring old commands for target `Modules/_math.o'
Makefile:2251: warning: overriding commands for target `Modules/posixmodule.o'
Makefile:2193: warning: ignoring old commands for target `Modules/posixmodule.o'
Makefile:2252: warning: overriding commands for target `Modules/posix.cpython-310-darwin.so'
Makefile:2194: warning: ignoring old commands for target `Modules/posix.cpython-310-darwin.so'
Makefile:2253: warning: overriding commands for target `Modules/errnomodule.o'
Makefile:2179: warning: ignoring old commands for target `Modules/errnomodule.o'
Makefile:2254: warning: overriding commands for target `Modules/errno.cpython-310-darwin.so'
Makefile:2180: warning: ignoring old commands for target `Modules/errno.cpython-310-darwin.so'
Makefile:2255: warning: overriding commands for target `Modules/pwdmodule.o'
Makefile:2195: warning: ignoring old commands for target `Modules/pwdmodule.o'
Makefile:2256: warning: overriding commands for target `Modules/pwd.cpython-310-darwin.so'
Makefile:2196: warning: ignoring old commands for target `Modules/pwd.cpython-310-darwin.so'
Makefile:2257: warning: overriding commands for target `Modules/_sre.o'
Makefile:2148: warning: ignoring old commands for target `Modules/_sre.o'
Makefile:2258: warning: overriding commands for target `Modules/_sre.cpython-310-darwin.so'
Makefile:2149: warning: ignoring old commands for target `Modules/_sre.cpython-310-darwin.so'
Makefile:2259: warning: overriding commands for target `Modules/_codecsmodule.o'
Makefile:2058: warning: ignoring old commands for target `Modules/_codecsmodule.o'
Makefile:2260: warning: overriding commands for target `Modules/_codecs.cpython-310-darwin.so'
Makefile:2059: warning: ignoring old commands for target `Modules/_codecs.cpython-310-darwin.so'
Makefile:2261: warning: overriding commands for target `Modules/_weakref.o'
Makefile:2164: warning: ignoring old commands for target `Modules/_weakref.o'
Makefile:2262: warning: overriding commands for target `Modules/_weakref.cpython-310-darwin.so'
Makefile:2165: warning: ignoring old commands for target `Modules/_weakref.cpython-310-darwin.so'
Makefile:2263: warning: overriding commands for target `Modules/_functoolsmodule.o'
Makefile:2084: warning: ignoring old commands for target `Modules/_functoolsmodule.o'
Makefile:2264: warning: overriding commands for target `Modules/_functools.cpython-310-darwin.so'
Makefile:2085: warning: ignoring old commands for target `Modules/_functools.cpython-310-darwin.so'
Makefile:2265: warning: overriding commands for target `Modules/_operator.o'
Makefile:2116: warning: ignoring old commands for target `Modules/_operator.o'
Makefile:2266: warning: overriding commands for target `Modules/_operator.cpython-310-darwin.so'
Makefile:2117: warning: ignoring old commands for target `Modules/_operator.cpython-310-darwin.so'
Makefile:2267: warning: overriding commands for target `Modules/_collectionsmodule.o'
Makefile:2072: warning: ignoring old commands for target `Modules/_collectionsmodule.o'
Makefile:2268: warning: overriding commands for target `Modules/_collections.cpython-310-darwin.so'
Makefile:2073: warning: ignoring old commands for target `Modules/_collections.cpython-310-darwin.so'
Makefile:2269: warning: overriding commands for target `Modules/_abc.o'
Makefile:2046: warning: ignoring old commands for target `Modules/_abc.o'
Makefile:2270: warning: overriding commands for target `Modules/_abc.cpython-310-darwin.so'
Makefile:2047: warning: ignoring old commands for target `Modules/_abc.cpython-310-darwin.so'
Makefile:2271: warning: overriding commands for target `Modules/itertoolsmodule.o'
Makefile:2187: warning: ignoring old commands for target `Modules/itertoolsmodule.o'
Makefile:2272: warning: overriding commands for target `Modules/itertools.cpython-310-darwin.so'
Makefile:2188: warning: ignoring old commands for target `Modules/itertools.cpython-310-darwin.so'
Makefile:2273: warning: overriding commands for target `Modules/atexitmodule.o'
Makefile:2172: warning: ignoring old commands for target `Modules/atexitmodule.o'
Makefile:2274: warning: overriding commands for target `Modules/atexit.cpython-310-darwin.so'
Makefile:2173: warning: ignoring old commands for target `Modules/atexit.cpython-310-darwin.so'
Makefile:2275: warning: overriding commands for target `Modules/signalmodule.o'
Makefile:2134: warning: ignoring old commands for target `Modules/signalmodule.o'
Makefile:2276: warning: overriding commands for target `Modules/_signal.cpython-310-darwin.so'
Makefile:2135: warning: ignoring old commands for target `Modules/_signal.cpython-310-darwin.so'
Makefile:2277: warning: overriding commands for target `Modules/_stat.o'
Makefile:2152: warning: ignoring old commands for target `Modules/_stat.o'
Makefile:2278: warning: overriding commands for target `Modules/_stat.cpython-310-darwin.so'
Makefile:2153: warning: ignoring old commands for target `Modules/_stat.cpython-310-darwin.so'
Makefile:2279: warning: overriding commands for target `Modules/timemodule.o'
Makefile:2210: warning: ignoring old commands for target `Modules/timemodule.o'
Makefile:2280: warning: overriding commands for target `Modules/time.cpython-310-darwin.so'
Makefile:2211: warning: ignoring old commands for target `Modules/time.cpython-310-darwin.so'
Makefile:2281: warning: overriding commands for target `Modules/_threadmodule.o'
Makefile:2160: warning: ignoring old commands for target `Modules/_threadmodule.o'
Makefile:2282: warning: overriding commands for target `Modules/_thread.cpython-310-darwin.so'
Makefile:2161: warning: ignoring old commands for target `Modules/_thread.cpython-310-darwin.so'
Makefile:2283: warning: overriding commands for target `Modules/_localemodule.o'
Makefile:2100: warning: ignoring old commands for target `Modules/_localemodule.o'
Makefile:2284: warning: overriding commands for target `Modules/_locale.cpython-310-darwin.so'
Makefile:2101: warning: ignoring old commands for target `Modules/_locale.cpython-310-darwin.so'
Makefile:2285: warning: overriding commands for target `Modules/_iomodule.o'
Makefile:2090: warning: ignoring old commands for target `Modules/_iomodule.o'
Makefile:2286: warning: overriding commands for target `Modules/iobase.o'
Makefile:2091: warning: ignoring old commands for target `Modules/iobase.o'
Makefile:2287: warning: overriding commands for target `Modules/fileio.o'
Makefile:2092: warning: ignoring old commands for target `Modules/fileio.o'
Makefile:2288: warning: overriding commands for target `Modules/bytesio.o'
Makefile:2093: warning: ignoring old commands for target `Modules/bytesio.o'
Makefile:2289: warning: overriding commands for target `Modules/bufferedio.o'
Makefile:2094: warning: ignoring old commands for target `Modules/bufferedio.o'
Makefile:2290: warning: overriding commands for target `Modules/textio.o'
Makefile:2095: warning: ignoring old commands for target `Modules/textio.o'
Makefile:2291: warning: overriding commands for target `Modules/stringio.o'
Makefile:2096: warning: ignoring old commands for target `Modules/stringio.o'
Makefile:2292: warning: overriding commands for target `Modules/_io.cpython-310-darwin.so'
Makefile:2097: warning: ignoring old commands for target `Modules/_io.cpython-310-darwin.so'
Makefile:2293: warning: overriding commands for target `Modules/faulthandler.o'
Makefile:2181: warning: ignoring old commands for target `Modules/faulthandler.o'
Makefile:2294: warning: overriding commands for target `Modules/faulthandler.cpython-310-darwin.so'
Makefile:2182: warning: ignoring old commands for target `Modules/faulthandler.cpython-310-darwin.so'
Makefile:2295: warning: overriding commands for target `Modules/_tracemalloc.o'
Makefile:2162: warning: ignoring old commands for target `Modules/_tracemalloc.o'
Makefile:2296: warning: overriding commands for target `Modules/_tracemalloc.cpython-310-darwin.so'
Makefile:2163: warning: ignoring old commands for target `Modules/_tracemalloc.cpython-310-darwin.so'
Makefile:2297: warning: overriding commands for target `Modules/symtablemodule.o'
Makefile:2158: warning: ignoring old commands for target `Modules/symtablemodule.o'
Makefile:2298: warning: overriding commands for target `Modules/_symtable.cpython-310-darwin.so'
Makefile:2159: warning: ignoring old commands for target `Modules/_symtable.cpython-310-darwin.so'
xcrun --sdk macosx clang --sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -mmacosx-version-min=10.15 -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64    -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include    -DPy_BUILD_CORE_BUILTIN  -I./../Support/OpenSSL.xcframework/macos-arm64_x86_64/Headers -DUSE_SSL -c ./Modules/_hashopenssl.c -o Modules/_hashopenssl.o
./Modules/_hashopenssl.c:33:10: fatal error: 'openssl/evp.h' file not found

freakboy3742 commented 2 years ago

Thanks for the report. Can you clarify the exact steps for reproducing this error? In particular:

"a very recent tag" doesn't tell us which tag that actually is, or how you obtained the source code - the paths suggest it might be a zip file, but it's not clear where you got that zip file from
Could you please provide the full build log (from clean), not just the "selected readings that look important".
Could I also get you to provide all the .log files contained in the build/macOS folder, and a (recursive) listing of the contents of the build/macOS/Support folder.

The logs you have provided suggest that either OpenSSL hasn't been compiled, or that something is being messed up in the construction of paths. However, the 3.10-b2 build ran through CI yesterday; and I've just done a clean run on git checkout 3730a2a95, and it worked fine, so I'm not sure where it's going off the rails.

riccardodivirgilio commented 2 years ago

Ok I've done the procedure again. I downloaded the repo at this commit, which should be tag 3.10, not 3.10-b2:

https://github.com/beeware/Python-Apple-support/archive/53e554e12b564e67ee7b49d5a5998c2bfce89341.zip

This is the output of the command.

stderr.log

This is the compressed archive with all folders under build/macOS

build-macOS-logs.zip

The directories under build/macOS/support seems to be empty 🤔?

3 directories, 0 files
testarmmacmini4:Python-Apple-support-53e554e12b564e67ee7b49d5a5998c2bfce89341 testmngr$ tree build/macOS/Support -f
build/macOS/Support
├── build/macOS/Support/BZip2.xcframework
├── build/macOS/Support/OpenSSL.xcframework
└── build/macOS/Support/XZ.xcframework

riccardodivirgilio commented 2 years ago

I have also tried to build using the commit you suggested, but it seems to fail in the very same way.

freakboy3742 commented 2 years ago

Ok - the repository has moved on substantially since that tag; it's probable that the problem you're seeing has been resolved. I'd suggest that you should update to a formally tagged release, and start there.

riccardodivirgilio commented 2 years ago

I've tried again using 3.10 branch:

wget https://github.com/beeware/Python-Apple-support/archive/refs/heads/3.10.zip

and 3.10-b2 tag:

wget https://github.com/beeware/Python-Apple-support/archive/refs/tags/3.10-b2.zip

they fail in the same way. Am I missing something? should I use another branch/tag?

Thanks.

freakboy3742 commented 2 years ago

Ah - now I see it. You're using the Command Line tools. You need to use a full Xcode install. The build process uses xcodebuild to construct the XCframework.

beeware / Python-Apple-support

build failure on latest 3.10 tag #147