search

aws / aws-sdk-cpp

AWS SDK for C++
Apache License 2.0
1.91k stars 1.04k forks source link

Build error with VS 2022, and zlib compression enabled #2422

Open jzakrzewski opened 1 year ago

jzakrzewski commented 1 year ago

Describe the bug

Build failed

Expected Behavior

Successful build

Current Behavior

FAILED: _deps/aws-sdk-build/src/aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/ub_core.cpp.obj 

C:\PROGRA~1\MIB055~1\2022\ENTERP~1\VC\Tools\MSVC\1435~1.322\bin\Hostx64\x64\cl.exe  /nologo /TP -DAWS_AUTH_USE_IMPORT_EXPORT -DAWS_CAL_USE_IMPORT_EXPORT -DAWS_CHECKSUMS_USE_IMPORT_EXPORT -DAWS_COMMON_USE_IMPORT_EXPORT -DAWS_COMPRESSION_USE_IMPORT_EXPORT -DAWS_CORE_EXPORTS=1 -DAWS_CRT_CPP_USE_IMPORT_EXPORT -DAWS_EVENT_STREAM_USE_IMPORT_EXPORT -DAWS_HTTP_USE_IMPORT_EXPORT -DAWS_IO_USE_IMPORT_EXPORT -DAWS_MQTT_USE_IMPORT_EXPORT -DAWS_MQTT_WITH_WEBSOCKETS -DAWS_S3_USE_IMPORT_EXPORT -DAWS_SDKUTILS_USE_IMPORT_EXPORT -DAWS_SDK_VERSION_MAJOR=1 -DAWS_SDK_VERSION_MINOR=11 -DAWS_SDK_VERSION_PATCH=48 -DAWS_TEST_REGION=US_EAST_1 -DAWS_USE_IO_COMPLETION_PORTS -DBOOST_ALL_NO_LIB -DCONSTEXPR=constexpr -DENABLED_REQUEST_COMPRESSION -DENABLED_ZLIB_REQUEST_COMPRESSION -DENABLE_BCRYPT_ENCRYPTION -DENABLE_CURL_LOGGING -DENABLE_WINDOWS_CLIENT -DFORCE_USE_RESULT="" -DJAVA_SCRIPT_DEBUG -DNOEXCEPT=noexcept -DPLATFORM_WINDOWS -DUSE_IMPORT_EXPORT=1 -DUSE_WINDOWS_DLL_SEMANTICS -DWINHTTP_HAS_H2 -DWININET_HAS_H2 -D_CRTDBG_MAP_ALLOC -D_HAS_AUTO_PTR_ETC=1 -Daws_cpp_sdk_core_EXPORTS -IC:\build -IC:\build\external\include -IC:\build\_deps\aws-sdk-src\src\aws-cpp-sdk-core\include\aws\core\platform\refs -IC:\build\_deps\aws-sdk-src\src\aws-cpp-sdk-core\include -IC:\build\_deps\aws-sdk-src\crt\aws-crt-cpp\include -IC:\build\_deps\aws-sdk-src\crt\aws-crt-cpp\crt\aws-c-http\include -IC:\build\_deps\aws-sdk-src\crt\aws-crt-cpp\crt\aws-c-io\include -IC:\build\_deps\aws-sdk-src\crt\aws-crt-cpp\crt\aws-c-common\include -IC:\build\_deps\aws-sdk-build\crt\aws-crt-cpp\crt\aws-c-common\generated\include -IC:\build\_deps\aws-sdk-src\crt\aws-crt-cpp\crt\aws-c-cal\include -IC:\build\_deps\aws-sdk-src\crt\aws-crt-cpp\crt\aws-c-compression\include -IC:\build\_deps\aws-sdk-src\crt\aws-crt-cpp\crt\aws-c-mqtt\include -IC:\build\_deps\aws-sdk-src\crt\aws-crt-cpp\crt\aws-c-auth\include -IC:\build\_deps\aws-sdk-src\crt\aws-crt-cpp\crt\aws-c-sdkutils\include -IC:\build\_deps\aws-sdk-src\crt\aws-crt-cpp\crt\aws-checksums\include -IC:\build\_deps\aws-sdk-src\crt\aws-crt-cpp\crt\aws-c-event-stream\include -IC:\build\_deps\aws-sdk-src\crt\aws-crt-cpp\crt\aws-c-s3\include -external:W0 /DWIN32 /D_WINDOWS /EHsc /MP /bigobj /utf-8 /WX /W4 /Zi /Ob0 /Od /RTC1 -MDd -FS -std:c++20 /showIncludes /Fo_deps\aws-sdk-build\src\aws-cpp-sdk-core\CMakeFiles\aws-cpp-sdk-core.dir\ub_core.cpp.obj /Fd_deps\aws-sdk-build\src\aws-cpp-sdk-core\CMakeFiles\aws-cpp-sdk-core.dir\ /FS -c C:\build\_deps\aws-sdk-build\src\aws-cpp-sdk-core\ub_core.cpp

C:/build/_deps/aws-sdk-src/src/aws-cpp-sdk-core/source/client/RequestCompression.cpp(159): error C2220: the following warning is treated as an error

C:/build/_deps/aws-sdk-src/src/aws-cpp-sdk-core/source/client/RequestCompression.cpp(159): warning C4267: '=': conversion from 'size_t' to 'uInt', possible loss of data

C:/build/_deps/aws-sdk-src/src/aws-cpp-sdk-core/source/client/RequestCompression.cpp(280): warning C4267: '=': conversion from 'size_t' to 'uInt', possible loss of data

Reproduction Steps

Build with Visual Studio 2022, and zlib compression enabled.

Possible Solution

In src/aws-cpp-sdk-core/source/client/RequestCompression.cpp:

line 159: strm.avail_in = static_cast<uInt>((flush == Z_FINISH)?toRead-1:toRead); //skip EOF if included

line 208: strm.avail_in = static_cast<uInt>(toRead);

Additional Information/Context

No response

AWS CPP SDK version used

1.11.48

Compiler and Version used

VS 2022

Operating System and version

Windows 10

jmklix commented 1 year ago

What build commands are you using to repro this?

jzakrzewski commented 1 year ago

Originally I used it as a submodule (via FetchContent), but I have just reproduced it stand-alone. This is the command line:

cmake -G Ninja -DCMAKE_BUILD_TYPE=Debug -DCPP_STANDARD=20 -DBUILD_ONLY=s3 -DENABLE_TESTING=OFF -DBUILD_TESTING=OFF -DAUTORUN_UNIT_TESTS=OFF -DENABLE_UNITY_BUILD=ON -DFORCE_CURL=ON -DUSE_OPENSSL=ON -DENABLE_ZLIB_REQUEST_COMPRESSION=ON -DCMAKE_PREFIX_PATH:PATH=C:\tmp\libs\debug -DCMAKE_INSTALL_PREFIX:PATH=C:\tmp\aws-install C:\tmp\aws-sdk-cpp-1.11.48

and the build:

c:\tmp\aws-build>ninja
[293/314] Building CXX object src\aws-cpp-sdk-core\CMakeFiles\aws-cpp-sdk-core.dir\ub_core.cpp.obj
FAILED: src/aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/ub_core.cpp.obj
C:\PROGRA~1\MIB055~1\2022\ENTERP~1\VC\Tools\MSVC\1435~1.322\bin\Hostx64\x64\cl.exe /nologo /TP -DAWS_AUTH_USE_IMPORT_EXPORT -DAWS_CAL_USE_IMPORT_EXPORT -DAWS_CHECKSUMS_USE_IMPORT_EXPORT -DAWS_COMMON_USE_IMPORT_EXPORT -DAWS_COMPRESSION_USE_IMPORT_EXPORT -DAWS_CORE_EXPORTS=1 -DAWS_CRT_CPP_USE_IMPORT_EXPORT -DAWS_EVENT_STREAM_USE_IMPORT_EXPORT -DAWS_HTTP_USE_IMPORT_EXPORT -DAWS_IO_USE_IMPORT_EXPORT -DAWS_MQTT_USE_IMPORT_EXPORT -DAWS_MQTT_WITH_WEBSOCKETS -DAWS_S3_USE_IMPORT_EXPORT -DAWS_SDKUTILS_USE_IMPORT_EXPORT -DAWS_SDK_VERSION_MAJOR=1 -DAWS_SDK_VERSION_MINOR=11 -DAWS_SDK_VERSION_PATCH=48 -DAWS_TEST_REGION=US_EAST_1 -DAWS_USE_IO_COMPLETION_PORTS -DENABLED_REQUEST_COMPRESSION -DENABLED_ZLIB_REQUEST_COMPRESSION -DENABLE_BCRYPT_ENCRYPTION -DENABLE_CURL_CLIENT -DENABLE_CURL_LOGGING -DPLATFORM_WINDOWS -DUSE_IMPORT_EXPORT=1 -DUSE_WINDOWS_DLL_SEMANTICS -Daws_cpp_sdk_core_EXPORTS -IC:\tmp\aws-sdk-cpp-1.11.48\src\aws-cpp-sdk-core\include\aws\core\platform\refs -IC:\tmp\aws-sdk-cpp-1.11.48\src\aws-cpp-sdk-core\include -IC:\tmp\aws-sdk-cpp-1.11.48\crt\aws-crt-cpp\include -IC:\tmp\aws-sdk-cpp-1.11.48\crt\aws-crt-cpp\crt\aws-c-http\include -IC:\tmp\aws-sdk-cpp-1.11.48\crt\aws-crt-cpp\crt\aws-c-io\include -IC:\tmp\aws-sdk-cpp-1.11.48\crt\aws-crt-cpp\crt\aws-c-common\include -IC:\tmp\aws-build\crt\aws-crt-cpp\crt\aws-c-common\generated\include -IC:\tmp\aws-sdk-cpp-1.11.48\crt\aws-crt-cpp\crt\aws-c-cal\include -IC:\tmp\aws-sdk-cpp-1.11.48\crt\aws-crt-cpp\crt\aws-c-compression\include -IC:\tmp\aws-sdk-cpp-1.11.48\crt\aws-crt-cpp\crt\aws-c-mqtt\include -IC:\tmp\aws-sdk-cpp-1.11.48\crt\aws-crt-cpp\crt\aws-c-auth\include -IC:\tmp\aws-sdk-cpp-1.11.48\crt\aws-crt-cpp\crt\aws-c-sdkutils\include -IC:\tmp\aws-sdk-cpp-1.11.48\crt\aws-crt-cpp\crt\aws-checksums\include -IC:\tmp\aws-sdk-cpp-1.11.48\crt\aws-crt-cpp\crt\aws-c-event-stream\include -IC:\tmp\aws-sdk-cpp-1.11.48\crt\aws-crt-cpp\crt\aws-c-s3\include -external:IC:\tmp\libs\debug\include -external:W0 /DWIN32 /D_WINDOWS /W4 /GR /EHsc /MP /bigobj /utf-8 /WX /MDd /Zi /Ob0 /Od /RTC1 /showIncludes /Fosrc\aws-cpp-sdk-core\CMakeFiles\aws-cpp-sdk-core.dir\ub_core.cpp.obj /Fdsrc\aws-cpp-sdk-core\CMakeFiles\aws-cpp-sdk-core.dir\ /FS -c C:\tmp\aws-build\src\aws-cpp-sdk-core\ub_core.cpp
C:/tmp/aws-sdk-cpp-1.11.48/src/aws-cpp-sdk-core/source/client/RequestCompression.cpp(159): error C2220: the following warning is treated as an error
C:/tmp/aws-sdk-cpp-1.11.48/src/aws-cpp-sdk-core/source/client/RequestCompression.cpp(159): warning C4267: '=': conversion from 'size_t' to 'uInt', possible loss of data
C:/tmp/aws-sdk-cpp-1.11.48/src/aws-cpp-sdk-core/source/client/RequestCompression.cpp(280): warning C4267: '=': conversion from 'size_t' to 'uInt', possible loss of data
[310/314] Building CXX object generated\src\aws-cpp-sdk-s3\CMakeFiles\aws-cpp-sdk-s3.dir\ub_S3.cpp.obj
ninja: build stopped: subcommand failed.
jzakrzewski commented 1 year ago

Any news?

radwaverf commented 12 months ago

I reproduced this error as well, but the fix may also require changing 

static const size_t ZLIB_CHUNK=263144;

to

static const size_t ZLIB_CHUNK=65536;

because uInt could only be 16 bits on some systems. My system has 32 bit unsigned int though, so I don't know how to test that case. Any ideas?

jzakrzewski commented 12 months ago

Hmmm - why does ZLIB_CHUNK have to be a size_t anyway? Maybe it should just be uInt? Maybe even initialized like: static const auto ZLIB_CHUNK=std::numeric_limits<uInt>::max();?

radwaverf commented 12 months ago

This is a bit beyond my knowledge of zlib, but maybe there are performance implications with ZLIB_CHUNK sizes? It seems like the only reason it's size_t is to match the type of streamSize, which could exceed 4 GB. So whether ZLIB_CHUNK is uInt or size_t, I think static_cast would be needed.

Also, 263144 isn't actually 256k. 256k is 262144. Is this a typo? https://github.com/aws/aws-sdk-cpp/blob/main/src/aws-cpp-sdk-core/source/client/RequestCompression.cpp#L22-L23 Or is that intentional so that when it rolls over with 16 bit systems it still a non-zero value?

AndyHUI711 commented 11 months ago

I got the same issue, but how to slove this

jzakrzewski commented 11 months ago

For now, I just patch the source (see the "Possible solution" section in issue description). I guess one could also build without compression support (for me it's a no-go).

Those are only workarounds. The actual solution must be provided by the authors. It makes me wonder, how they build it...

RochaStratovan commented 5 months ago

Almost a year old? Any official fixes for this yet?

jmklix commented 1 month ago

Can you try following this guide for building this sdk in Visual Studio? Please let me know if zlib compression is still causing any errors for you

github-actions[bot] commented 1 month ago

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

jzakrzewski commented 1 month ago

Sorry, but I cannot just jump into it. I had more than enough time when I first opened the issue, but nobody was interested back then...