[Regression] RPCS3 no longer builds

[Shoegzer]

Quick summary

RPCS3 no longer builds, at least as observed under Linux with Clang-17.

Details

The build issue appears to be related to curl. From a log excerpt:

[1836/2757] Building C object 3rdparty/curl/curl/lib/CMakeFiles/libcurl_static.dir/http_aws_sigv4.c.o
FAILED: 3rdparty/curl/curl/lib/CMakeFiles/libcurl_static.dir/http_aws_sigv4.c.o 
...
http_aws_sigv4.c.o -c /home/testing/Desktop/rpcs3/rpcs3/3rdparty/curl/curl/lib/http_aws_sigv4.c
/home/testing/Desktop/rpcs3/rpcs3/3rdparty/curl/curl/lib/http_aws_sigv4.c:68:40: error: use of undeclared identifier 'SHA256_DIGEST_LENGTH'
   68 |                  (unsigned char *)dst, SHA256_HEX_LENGTH);

Build with regression

d6380a1084e82a83b93917bed2ec1d8ed3c84ffb

Attach two log files

build.log

Attach capture files for visual issues

N/A

System configuration

• OS: Linux Mint 21.2 / kernel 6.5
• Compiler: Clang-17

Other details

The above regressed commit is a best guess, given that it's the only commit that changed anything curl-related. I have not yet had an opportunity to bisect but I will if necessary.

Last good build: a1d2a72a78fdd858248c4371c2aa08be4f612c20 Build with issue: 500bf0f3f5422b31ea4c2b70e2b0b19329deac7f

[StatusCode404]

No longer builds for me either but I have a different error in the logs...

In file included from /home/danglingpointer/Software/RPCS3/rpcs3/rpcs3/Emu/Cell/lv2/sys_usbd.cpp:20:
/home/danglingpointer/Software/RPCS3/rpcs3/rpcs3/Emu/Io/Dimensions.h:28:7: error: no template named 'optional' in namespace 'std'
   28 |         std::optional<std::array<u8, 32>> pop_added_removed_response();
      |         ~~~~~^
/home/danglingpointer/Software/RPCS3/rpcs3/rpcs3/Emu/Io/Dimensions.h:42:48: error: no template named 'optional' in namespace 'std'
   42 |         std::array<u8, 8> decrypt(const u8* buf, std::optional<std::array<u8, 16>> key);
      |                                                  ~~~~~^
/home/danglingpointer/Software/RPCS3/rpcs3/rpcs3/Emu/Io/Dimensions.h:43:48: error: no template named 'optional' in namespace 'std'
   43 |         std::array<u8, 8> encrypt(const u8* buf, std::optional<std::array<u8, 16>> key);
      |                                                  ~~~~~^
3 errors generated.
make[2]: *** [rpcs3/Emu/CMakeFiles/rpcs3_emu.dir/build.make:2008: rpcs3/Emu/CMakeFiles/rpcs3_emu.dir/Cell/lv2/sys_usbd.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:3114: rpcs3/Emu/CMakeFiles/rpcs3_emu.dir/all] Error 2
make: *** [Makefile:156: all] Error 2

I'm using Ubuntu clang version 17.0.6 build command...

time (git submodule update --init && cd .. && rm -rf rpcs3_build && mkdir --parents rpcs3_build && cd rpcs3_build && cmake ../rpcs3/ -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_AR="/usr/bin/llvm-ar" -DCMAKE_NM="/usr/bin/llvm-nm" -DCMAKE_RANLIB="/usr/bin/llvm-ranlib" -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_FLAGS_RELEASE="-Ofast -funroll-loops -march=native -mtune=native -pipe -DNDEBUG" -DCMAKE_CXX_FLAGS_RELEASE="-Ofast -funroll-loops -march=native -mtune=native -pipe -DNDEBUG" -DCMAKE_ASM_FLAGS_RELEASE="-Ofast -funroll-loops -march=native -mtune=native -pipe -DNDEBUG" -DCMAKE_PREFIX_PATH="/mnt/btrfsSSD/Software/QT6.6/6.6.3/gcc_64/;/mnt/btrfsSSD/Software/SDL2/installLocation" -DUSE_NATIVE_INSTRUCTIONS=ON -DUSE_PULSE=ON -DUSE_ALSA=ON && make -j $(nproc))

[StatusCode404]

Would be good if we had release tags that were known to work. Production style.

[StatusCode404]

Builds OK/Successfully with GCC-14.1 (custom default in my Ubuntu 22.04 LTS)

Testing now... .... working fine.

build command is similar to above minus the LLVM/CLang -DCMAKE options.

[RipleyTom]

Compiles here with clang version 18.1.8 on current master with no issue. Are you sure you updated submodules properly and you are doing a clean build(ie fully wipe build directory before building)?

[Shoegzer]

@StatusCode404 thanks for testing. Be aware that building with gcc does have its own issues though, at least with gcc-13.

@RipleyTom interesting, perhaps there's an issue with earlier clang versions? And yes I'm starting from a clean build (new source and build dirs etc.) and using git clone --recursive https://github.com/RPCS3/rpcs3.git. Also building with Ninja in case it matters.

[StatusCode404]

Thanks for responding @RipleyTom I believe I'm updating them correctly, I pasted my init, configure, and build command in my clang-17 build above. Just to re-quote here anyway I believe this should do the trick you were suggesting right?.... ....git submodule update --init && cd .. && rm -rf rpcs3_build....

@Shoegzer yeah thanks for the heads up, I've notice that with some games, if I remember correctly, I lost audio that was with gcc-13.2 when I tried it back then. It was a hit and miss with some games not having audio.

[hrzhu]

I have this SHA256_DIGEST_LENGTH compilation error on Kubuntu 24.04 as well. It happens with gcc 13.2, clang 18.1.3 from Ubuntu repo and my self-compiled gcc 14. The workaround is to manually revert wolfssl to v5.7.0-stable.

[oltolm]

What does 3rdparty/wolfssl/wolfssl/wolfssl/options.h look like in the build directory? It's the only other place where OPENSSL_EXTRA is defined beside the command like. Also make sure you have no 3rdparty/wolfssl/wolfssl/wolfssl/options.h in your source directory.

[AniLeo]

Needs retesting after #15816

[hrzhu]

https://github.com/RPCS3/rpcs3/pull/15816 maybe fix StatusCode404's problem, but not OP's SHA256_DIGEST_LENGTH problem.

[Shoegzer]

@oltolm:

rpcs3/build/3rdparty/wolfssl/wolfssl/wolfssl/options.h: options.h.zip (zipped since GH does't permit uploading files with .h extensions) The only reference to OPENSSL_EXTRA is: #undef OPENSSL_EXTRA

Also there is no rpcs3/3rdparty/wolfssl/wolfssl/wolfssl/options.h, only options.h.in

@AniLeo tested with latest master, confimed that the SHA256_DIGEST_LENGTH issue remains.

[oltolm]

@oltolm:

rpcs3/build/3rdparty/wolfssl/wolfssl/wolfssl/options.h: options.h.zip (zipped since GH does't permit uploading files with .h extensions) The only reference to OPENSSL_EXTRA is: #undef OPENSSL_EXTRA

That's the problem. It's not being properly generated by CMake.

[Shoegzer]

Yep, so an issue with CMake? This test system is using cmake 3.28.3, within the current requirement for 3.28.0.

[oltolm]

I don't know why it happens. You could try uncommenting the commented lines 2340 and 2343 in 3rdparty\wolfssl\wolfssl\CMakeLists.txt to enable message logging. Like this

foreach(DEF IN LISTS WOLFSSL_DEFINITIONS)
    string(REGEX MATCH "^(-D)?([^=]+)(=(.*))?$" DEF_MATCH ${DEF})
    if (DEFINED CMAKE_MATCH_4)
        set(${CMAKE_MATCH_2} ${CMAKE_MATCH_4})
        message("set(${CMAKE_MATCH_2} ${CMAKE_MATCH_4})")
    else()
        set(${CMAKE_MATCH_2} 1)
        message("set(${CMAKE_MATCH_2} 1)")
    endif()
endforeach()

and rerun CMake. Maybe delete 3rdparty/wolfssl/wolfssl/wolfssl/options.h first. I have never experienced such a problem.

[StatusCode404]

cmake --version
cmake version 3.29.6

[StatusCode404]

@hrzhu interesting that your GCC-14 failed to build. Mine builds fine with 14.1 also self built GCC from source.

The only thing I changed between LLVM/clang 17 and GCC-14 was the removal of the -DCMAKE options for LLVM/Clang. I removed these...

-DCMAKE_C_COMPILER=clang 
-DCMAKE_CXX_COMPILER=clang++ 
-DCMAKE_AR="/usr/bin/llvm-ar" 
-DCMAKE_NM="/usr/bin/llvm-nm" 
-DCMAKE_RANLIB="/usr/bin/llvm-ranlib"

And builds ok with GCC-14.1 Full GCC cmake and build command...

time (git submodule update --init && cd .. && rm -rf rpcs3_build && mkdir --parents rpcs3_build && cd rpcs3_build && cmake ../rpcs3/ -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_FLAGS_RELEASE="-Ofast -funroll-loops -march=native -mtune=native -pipe -DNDEBUG" -DCMAKE_CXX_FLAGS_RELEASE="-Ofast -funroll-loops -march=native -mtune=native -pipe -DNDEBUG" -DCMAKE_ASM_FLAGS_RELEASE="-Ofast -funroll-loops -march=native -mtune=native -pipe -DNDEBUG" -DCMAKE_PREFIX_PATH="/mnt/btrfsSSD/Software/QT6.6/6.6.3/gcc_64/;/mnt/btrfsSSD/Software/SDL2/installLocation" -DUSE_NATIVE_INSTRUCTIONS=ON -DUSE_PULSE=ON -DUSE_ALSA=ON && make -j $(nproc))

I do have a local build of QT6.6.3 and SDL2. Not sure that is relevant though. I have referenced them via "-DCMAKE_PREFIX_PATH"

[hrzhu]

Another workaround is to build with -DUSE_SYSTEM_CURL=ON. That's what the CI build does.

[T2theV]

The system Curl flag worked in a ubuntu:jammy container I just used to build.

[Shoegzer]

@oltolm I uncommented those lines from 3rdparty\wolfssl\wolfssl\CMakeLists.txt and rebuilt from a fresh build directory (so no 3rdparty/wolfssl/wolfssl/wolfssl/options.h to delete). It didn't reveal much, and just one reference to OPENSSL_EXTRA: set(OPENSSL_EXTRA )

Here is the full logfile from this attempt: build_with_message_logging.log

@hrzhu I can also confirm that -DUSE_SYSTEM_CURL=ON works, though as you said it's a workaround. Although I had thought it was a cmake issue, it seems more related to an issue with newer releases of curl. I looked here but nothing came up in my searches, though I'm likely missing it as I'm not so certain this project would be seeing it first.

[Megamouse]

Please try again. I updated curl.

[StatusCode404]

I can confirm that it builds ok as well with just released GCC-14.2.

[hrzhu]

I still have this SHA256_DIGEST_LENGTH issue when compiling git master

[Shoegzer]

I can confirm that the latest master with the curl update still fails with SHA256_DIGEST_LENGTH errors. Building with gcc-14.0.1

[StatusCode404]

I can confirm that the latest master with the curl update still fails with SHA256_DIGEST_LENGTH errors. Building with gcc-14.0.1

I built using cmake version 3.29.6. Also see if you can get your hands on GCC-14.2. Try those.

The only other difference is that I have my own built SDL2 and QT6.6.3. You can try copying my build command that I pasted above to see if that works. Just readjust for your SDL2 and QT.

[Shoegzer]

Thanks. I kept my cmake at 3.20.3 and gcc to 14.0.1 (both same as above), but I rebuilt SDL2 2.30 and Qt 6.7, and it's compiling now. I had been using Qt 6.8 which is still in beta, so perhaps the downgrade to the stable release fixed this even though it doesn't seem likely. In any event I'll post the build log for reference in case it helps others.

@hrzhu Try to follow these versions and see if you can resolve your build issue too, it would be good to close this one.

rpcs3build.log

[oltolm]

I think I know what the problem is and I created a PR upstream.

[oltolm]

My PR has been merged upstream.

[Shoegzer]

Thanks @oltolm. I haven't heard from @hrzhu so I'm just going to close this, especially now that your PR has been merged and I'm not having difficulties building anymore.