dbjh
commented
6 months ago For ease of testing with a file larger than 4GB I have attached a gzipped file of 5,000,000,000 bytes (4,852,396 bytes compressed). The file consists entirely of bytes with value 0, except the byte at offset 0x100000000 (4GB), in order to help detect that all data is read (instead of internal file pointers of zlib wrapping around at 4GB).
The expected output of ./tell bigfile.gz with tell.c compiled with either MinGW or MinGW-w64 (GCC & Clang) and zlib 1.3.1 with my pull request applied is:
Using zlib version: 1.3.1 File position: 5000000000; sizeof (z_off64_t): 8
vszakats
sezero
madler
Problem
The issue causes the 64-bit functions to effectively use a 32-bit type, with the obvious consequences. For example, gztell64() will not return a correct value for (gzipped) files larger than 4GB if the file pointer has been moved beyond that point. The problems occurs with MinGW and MinGW-w64 (GCC & Clang). The 64-bit functions are only available when the defined constants _LARGEFILE64_SOURCE and _LFS64_LARGEFILE are non-zero. In zlib.h it is stated that
and
With MinGW-w64 the problem can be worked around by defining _FILE_OFFSET_BITS as 64 and defining {Z_}HAVE_UNISTD_H, but that does not match with those statements -- only defining _LARGEFILE64_SOURCE and _LFS64_LARGEFILE as non-zero should be enough.
Cause
In case of MinGW, z_off_t is always 32-bit and _FILE_OFFSET_BITS is ignored. The file zconf.h contains a test,
defined(_WIN32) && !defined(__GNUC__)to guard a reference to __int64. Perhaps based on the assumption that GNU C compilers do not support that type, while other Windows compilers do. That assumption is at least partially incorrect. To make that piece of preprocessor logic work correctly for MinGW, z_off64_t should be defined as __int64.Solution
As far as I know, all ports of GCC on Windows (i.e. defining _WIN32) are (part of):
Clang is not a port of GCC, but like GCC defines __GNUC__, usually. Except for Cygwin, they all support __int64 on Windows.
I realize there are more details regarding __GNUC__ on Windows. Clang may not always define it (https://blog.conan.io/2022/10/13/Different-flavors-Clang-compiler-Windows.html). If a compiler does not define __GNUC__, it either also supports __int64 or it cannot compile zlib in its current form (if Z_LARGE64=1 and/or _LARGEFILE64_SOURCE=1 and _LFS64_LARGEFILE=1). Clang as part of Cygwin does not support __int64. Clang as part of MinGW-w64 does.
So, although a positive test for compilers that support __int64 may be better, it is safer to stay as close to
!defined(__GNUC__)as possible, i.e., just expand the set of compilers that will refer to __int64 with the ones mentioned above. In my pull request I proposedefined(_WIN32) && !defined(__CYGWIN__). That solves the problem for MinGW and MinGW-w64 (GCC & Clang) and does not introduce a problem for Cygwin (GCC & Clang).Note that MinGW-w64 (GCC & Clang) does not solely depend on __int64, because there z_off_t can be 64-bit. However, as mentioned above this only happens when _FILE_OFFSET_BITS=64 and {Z_}HAVE_UNISTD_H defined, which should not be a requirement (similar to how it is no requirement with Cygwin or on 64-bit Linux).
The attached code tell.zip can be used to demonstrate the problem and as verification that the problem is not present or has been solved. Example:
tell file_larger_than_4gb.Update (2024-02-27)
I looked a bit deeper into support for __int64 by GCC and made the following findings:
#define __int64 long long).The above explains the behavior I described for Clang depending on the environment (MinGW-w64 versus Cygwin).
Availability of __int64 with GCC depends on inclusion of certain types of header file, which can be a problem (compilation error) if ZSOLO is defined. So, my fix required a test whether \_int64 is defined. That is why I changed:
&& !defined(__CYGWIN__)into:|| (defined(__GNUC__) && defined(__int64))There are of course numerous alternative ways to solve this issue, but this way may be the smallest modification. In the context of this issue zconf.h itself includes the standard C header file unistd.h, so for MinGW and MinGW-w64 availability of __int64 is guaranteed.If windows.h is included (zconf.h includes windows.h under certain conditions) __int64 is available with Cygwin too. Note that this issue doesn't occur with Cygwin.