search

microsoft / SPTAG

A distributed approximate nearest neighborhood search (ANN) library which provides a high quality vector index build, search and distributed online serving toolkits for large scale vector search scenario.
MIT License
4.83k stars 581 forks source link

build failure with GCC 13 due to missing `#include <cstdint> in `AnnService/inc/Helper/DiskIO.h` #389

Open pabs3 opened 1 year ago

pabs3 commented 1 year ago

Forwarding Debian sptag bug #1037866

Describe the bug SPTAG fails to build on Linux with GCC 13, because AnnService/inc/Helper/DiskIO.h uses std::uint16_t, std::uint32_t and std::uint64_t but does not #include <cstdint>.

To Reproduce Install GCC 13 and build SPTAG.

Steps to reproduce the behavior:

  1. Install Debian
  2. Upgrade to Debian 13
  3. Add unstable to apt sources
  4. apt update
  5. apt install gcc-13 g++-13
  6. Add experimental to apt sources
  7. apt update
  8. apt install -t experimental gcc g++
  9. Build SPTAG
  10. Notice failure

Expected behavior All files should include the headers they need to use the types, variables and functions that they use but don't define.

All of the following files probably should contain #include <cstdint>:

$ for f in `git grep -lE 'std::u?int[0-9]'` ; do git grep stdint $f || echo $f missing cstdint; done
AnnService/inc/Aggregator/AggregatorExecutionContext.h missing cstdint
AnnService/inc/Aggregator/AggregatorSettings.h missing cstdint
AnnService/inc/Client/ClientWrapper.h missing cstdint
AnnService/inc/Client/Options.h missing cstdint
AnnService/inc/Core/BKT/Index.h missing cstdint
AnnService/inc/Core/Common.h:#include <cstdint>
AnnService/inc/Core/Common/BKTree.h missing cstdint
AnnService/inc/Core/Common/Dataset.h missing cstdint
AnnService/inc/Core/Common/DistanceUtils.h missing cstdint
AnnService/inc/Core/Common/IQuantizer.h:#include <cstdint>
AnnService/inc/Core/Common/KDTree.h missing cstdint
AnnService/inc/Core/Common/Labelset.h missing cstdint
AnnService/inc/Core/Common/NeighborhoodGraph.h missing cstdint
AnnService/inc/Core/Common/OPQQuantizer.h missing cstdint
AnnService/inc/Core/Common/PQQuantizer.h missing cstdint
AnnService/inc/Core/Common/SIMDUtils.h missing cstdint
AnnService/inc/Core/CommonDataStructure.h missing cstdint
AnnService/inc/Core/DefinitionList.h missing cstdint
AnnService/inc/Core/KDT/Index.h missing cstdint
AnnService/inc/Core/MetadataSet.h missing cstdint
AnnService/inc/Core/SPANN/ExtraFullGraphSearcher.h missing cstdint
AnnService/inc/Core/SPANN/IExtraSearcher.h missing cstdint
AnnService/inc/Core/SPANN/Index.h missing cstdint
AnnService/inc/Core/SearchQuery.h missing cstdint
AnnService/inc/Core/VectorIndex.h missing cstdint
AnnService/inc/Helper/AsyncFileReader.h:#include <cstdint>
AnnService/inc/Helper/AsyncFileReader.h:#include <stdint.h>
AnnService/inc/Helper/Base64Encode.h:#include <cstdint>
AnnService/inc/Helper/Concurrent.h missing cstdint
AnnService/inc/Helper/DiskIO.h missing cstdint
AnnService/inc/Helper/LockFree.h:#include <cstdint>
AnnService/inc/Helper/StringConvert.h missing cstdint
AnnService/inc/Helper/VectorSetReader.h missing cstdint
AnnService/inc/Helper/VectorSetReaders/MemoryReader.h missing cstdint
AnnService/inc/Helper/VectorSetReaders/TxtReader.h missing cstdint
AnnService/inc/Server/SearchService.h missing cstdint
AnnService/inc/Socket/Client.h missing cstdint
AnnService/inc/Socket/Common.h:#include <cstdint>
AnnService/inc/Socket/Connection.h:#include <cstdint>
AnnService/inc/Socket/ConnectionManager.h:#include <cstdint>
AnnService/inc/Socket/Packet.h:#include <cstdint>
AnnService/inc/Socket/RemoteSearchQuery.h:#include <cstdint>
AnnService/inc/Socket/ResourceManager.h missing cstdint
AnnService/inc/Socket/SimpleSerialization.h:#include <cstdint>
AnnService/src/Aggregator/AggregatorContext.cpp missing cstdint
AnnService/src/Aggregator/AggregatorExecutionContext.cpp missing cstdint
AnnService/src/Aggregator/AggregatorService.cpp missing cstdint
AnnService/src/BalancedDataPartition/main.cpp missing cstdint
AnnService/src/Client/ClientWrapper.cpp missing cstdint
AnnService/src/Client/main.cpp missing cstdint
AnnService/src/Core/BKT/BKTIndex.cpp missing cstdint
AnnService/src/Core/Common/DistanceUtils.cpp missing cstdint
AnnService/src/Core/Common/IQuantizer.cpp missing cstdint
AnnService/src/Core/Common/SIMDUtils.cpp missing cstdint
AnnService/src/Core/KDT/KDTIndex.cpp missing cstdint
AnnService/src/Core/MetadataSet.cpp missing cstdint
AnnService/src/Core/SPANN/SPANNIndex.cpp missing cstdint
AnnService/src/Core/VectorIndex.cpp missing cstdint
AnnService/src/Helper/AsyncFileReader.cpp missing cstdint
AnnService/src/Helper/Base64Encode.cpp missing cstdint
AnnService/src/Helper/Concurrent.cpp missing cstdint
AnnService/src/Helper/SimpleIniReader.cpp missing cstdint
AnnService/src/Helper/VectorSetReader.cpp missing cstdint
AnnService/src/Helper/VectorSetReaders/DefaultReader.cpp missing cstdint
AnnService/src/Helper/VectorSetReaders/TxtReader.cpp missing cstdint
AnnService/src/Helper/VectorSetReaders/XvecReader.cpp missing cstdint
AnnService/src/IndexSearcher/main.cpp missing cstdint
AnnService/src/Quantizer/main.cpp missing cstdint
AnnService/src/Server/SearchService.cpp missing cstdint
AnnService/src/Server/ServiceContext.cpp missing cstdint
AnnService/src/Socket/Client.cpp missing cstdint
AnnService/src/Socket/ConnectionManager.cpp missing cstdint
AnnService/src/Socket/Packet.cpp missing cstdint
AnnService/src/Socket/RemoteSearchQuery.cpp missing cstdint
Test/src/AlgoTest.cpp missing cstdint
Test/src/ConcurrentTest.cpp missing cstdint
Test/src/DistanceTest.cpp missing cstdint
Test/src/FilterTest.cpp missing cstdint
Test/src/PerfTest.cpp missing cstdint
Test/src/ReconstructIndexSimilarityTest.cpp missing cstdint
Test/src/SIMDTest.cpp missing cstdint
Wrappers/WinRT/AnnIndex.cpp missing cstdint
Wrappers/inc/ClientInterface.h missing cstdint
Wrappers/inc/CsharpCommon.i missing cstdint
Wrappers/inc/JavaCommon.i missing cstdint
Wrappers/inc/PythonCommon.i missing cstdint
Wrappers/src/CLRCoreInterface.cpp missing cstdint
Wrappers/src/ClientInterface.cpp missing cstdint
Wrappers/src/CoreInterface.cpp missing cstdint

Screenshots

cd "build/AnnService" && /usr/bin/c++ -DNUMA -DSPTAGLib_EXPORTS -I"AnnService" -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wall -Wunreachable-code -Wno-reorder -Wno-delete-non-virtual-dtor -Wno-sign-compare -Wno-unknown-pragmas -Wcast-align -lm -lrt -fopenmp -lnuma -fopenmp -std=gnu++17 -fPIC -MD -MT AnnService/CMakeFiles/SPTAGLib.dir/src/Helper/AsyncFileReader.cpp.o -MF CMakeFiles/SPTAGLib.dir/src/Helper/AsyncFileReader.cpp.o.d -o CMakeFiles/SPTAGLib.dir/src/Helper/AsyncFileReader.cpp.o -c "AnnService/src/Helper/AsyncFileReader.cpp"
In file included from AnnService/inc/Helper/AsyncFileReader.h:7,
                 from AnnService/src/Helper/AsyncFileReader.cpp:4:
AnnService/inc/Helper/DiskIO.h:29:18: error: 'uint64_t' in namespace 'std' does not name a type; did you mean 'wint_t'?
   29 |             std::uint64_t m_offset;
      |                  ^~~~~~~~
      |                  wint_t
AnnService/inc/Helper/DiskIO.h:30:18: error: 'uint64_t' in namespace 'std' does not name a type; did you mean 'wint_t'?
   30 |             std::uint64_t m_readSize;
      |                  ^~~~~~~~
      |                  wint_t
AnnService/inc/Helper/DiskIO.h: In constructor 'SPTAG::Helper::AsyncReadRequest::AsyncReadRequest()':
AnnService/inc/Helper/DiskIO.h:42:34: error: class 'SPTAG::Helper::AsyncReadRequest' does not have any field named 'm_offset'
   42 |             AsyncReadRequest() : m_offset(0), m_readSize(0), m_buffer(nullptr), m_status(0), m_payload(nullptr), m_success(false), m_extension(nullptr) {}
      |                                  ^~~~~~~~
AnnService/inc/Helper/DiskIO.h:42:47: error: class 'SPTAG::Helper::AsyncReadRequest' does not have any field named 'm_readSize'
   42 |             AsyncReadRequest() : m_offset(0), m_readSize(0), m_buffer(nullptr), m_status(0), m_payload(nullptr), m_success(false), m_extension(nullptr) {}
      |                                               ^~~~~~~~~~
AnnService/inc/Helper/DiskIO.h: At global scope:
AnnService/inc/Helper/DiskIO.h:54:17: error: 'std::uint64_t' has not been declared
   54 |                 std::uint64_t maxIOSize = (1 << 20),
      |                 ^~~
AnnService/inc/Helper/DiskIO.h:55:17: error: 'std::uint32_t' has not been declared
   55 |                 std::uint32_t maxReadRetries = 2,
      |                 ^~~
AnnService/inc/Helper/DiskIO.h:56:17: error: 'std::uint32_t' has not been declared
   56 |                 std::uint32_t maxWriteRetries = 2,
      |                 ^~~
AnnService/inc/Helper/DiskIO.h:57:17: error: 'std::uint16_t' has not been declared
   57 |                 std::uint16_t threadPoolSize = 4) = 0;
      |                 ^~~
AnnService/inc/Helper/DiskIO.h:59:26: error: 'uint64_t' in namespace 'std' does not name a type; did you mean 'wint_t'?
   59 |             virtual std::uint64_t ReadBinary(std::uint64_t readSize, char* buffer, std::uint64_t offset = UINT64_MAX) = 0;
      |                          ^~~~~~~~
      |                          wint_t
AnnService/inc/Helper/DiskIO.h:61:26: error: 'uint64_t' in namespace 'std' does not name a type; did you mean 'wint_t'?
   61 |             virtual std::uint64_t WriteBinary(std::uint64_t writeSize, const char* buffer, std::uint64_t offset = UINT64_MAX) = 0;
      |                          ^~~~~~~~
      |                          wint_t
AnnService/inc/Helper/DiskIO.h:63:26: error: 'uint64_t' in namespace 'std' does not name a type; did you mean 'wint_t'?
   63 |             virtual std::uint64_t ReadString(std::uint64_t& readSize, std::unique_ptr<char[]>& buffer, char delim = '\n', std::uint64_t offset = UINT64_MAX) = 0;
      |                          ^~~~~~~~
      |                          wint_t
AnnService/inc/Helper/DiskIO.h:65:26: error: 'uint64_t' in namespace 'std' does not name a type; did you mean 'wint_t'?
   65 |             virtual std::uint64_t WriteString(const char* buffer, std::uint64_t offset = UINT64_MAX) = 0;
      |                          ^~~~~~~~
      |                          wint_t
AnnService/inc/Helper/DiskIO.h:69:72: error: 'std::uint32_t' has not been declared
   69 |             virtual bool BatchReadFile(AsyncReadRequest* readRequests, std::uint32_t requestCount) { return false; }
      |                                                                        ^~~
AnnService/inc/Helper/DiskIO.h:71:92: error: 'std::uint32_t' has not been declared
   71 |             virtual bool BatchCleanRequests(SPTAG::Helper::AsyncReadRequest* readRequests, std::uint32_t requestCount) { return false; }
      |                                                                                            ^~~
AnnService/inc/Helper/DiskIO.h:73:26: error: 'uint64_t' in namespace 'std' does not name a type; did you mean 'wint_t'?
   73 |             virtual std::uint64_t TellP() = 0;
      |                          ^~~~~~~~
      |                          wint_t
AnnService/inc/Helper/DiskIO.h:87:17: error: 'std::uint64_t' has not been declared
   87 |                 std::uint64_t maxIOSize = (1 << 20),
      |                 ^~~
AnnService/inc/Helper/DiskIO.h:88:17: error: 'std::uint32_t' has not been declared
   88 |                 std::uint32_t maxReadRetries = 2,
      |                 ^~~
AnnService/inc/Helper/DiskIO.h:89:17: error: 'std::uint32_t' has not been declared
   89 |                 std::uint32_t maxWriteRetries = 2,
      |                 ^~~
AnnService/inc/Helper/DiskIO.h:90:17: error: 'std::uint16_t' has not been declared
   90 |                 std::uint16_t threadPoolSize = 4)
      |                 ^~~
AnnService/inc/Helper/DiskIO.h:96:26: error: 'uint64_t' in namespace 'std' does not name a type; did you mean 'wint_t'?
   96 |             virtual std::uint64_t ReadBinary(std::uint64_t readSize, char* buffer, std::uint64_t offset = UINT64_MAX)
      |                          ^~~~~~~~
      |                          wint_t
AnnService/inc/Helper/DiskIO.h:103:26: error: 'uint64_t' in namespace 'std' does not name a type; did you mean 'wint_t'?
  103 |             virtual std::uint64_t WriteBinary(std::uint64_t writeSize, const char* buffer, std::uint64_t offset = UINT64_MAX)
      |                          ^~~~~~~~
      |                          wint_t
AnnService/inc/Helper/DiskIO.h:111:26: error: 'uint64_t' in namespace 'std' does not name a type; did you mean 'wint_t'?
  111 |             virtual std::uint64_t ReadString(std::uint64_t& readSize, std::unique_ptr<char[]>& buffer, char delim = '\n', std::uint64_t offset = UINT64_MAX)
      |                          ^~~~~~~~
      |                          wint_t
AnnService/inc/Helper/DiskIO.h:144:26: error: 'uint64_t' in namespace 'std' does not name a type; did you mean 'wint_t'?
  144 |             virtual std::uint64_t WriteString(const char* buffer, std::uint64_t offset = UINT64_MAX)
      |                          ^~~~~~~~
      |                          wint_t
AnnService/inc/Helper/DiskIO.h:149:26: error: 'uint64_t' in namespace 'std' does not name a type; did you mean 'wint_t'?
  149 |             virtual std::uint64_t TellP()
      |                          ^~~~~~~~
      |                          wint_t
AnnService/inc/Helper/DiskIO.h:176:22: error: 'uint64_t' in namespace 'std' does not name a type; did you mean 'wint_t'?
  176 |                 std::uint64_t tellp()
      |                      ^~~~~~~~
      |                      wint_t
AnnService/inc/Helper/DiskIO.h:192:17: error: 'std::uint64_t' has not been declared
  192 |                 std::uint64_t maxIOSize = (1 << 20),
      |                 ^~~
AnnService/inc/Helper/DiskIO.h:193:17: error: 'std::uint32_t' has not been declared
  193 |                 std::uint32_t maxReadRetries = 2,
      |                 ^~~
AnnService/inc/Helper/DiskIO.h:194:17: error: 'std::uint32_t' has not been declared
  194 |                 std::uint32_t maxWriteRetries = 2,
      |                 ^~~
AnnService/inc/Helper/DiskIO.h:195:17: error: 'std::uint16_t' has not been declared
  195 |                 std::uint16_t threadPoolSize = 4)
      |                 ^~~
AnnService/inc/Helper/DiskIO.h:204:26: error: 'uint64_t' in namespace 'std' does not name a type; did you mean 'wint_t'?
  204 |             virtual std::uint64_t ReadBinary(std::uint64_t readSize, char* buffer, std::uint64_t offset = UINT64_MAX)
      |                          ^~~~~~~~
      |                          wint_t
AnnService/inc/Helper/DiskIO.h:210:26: error: 'uint64_t' in namespace 'std' does not name a type; did you mean 'wint_t'?
  210 |             virtual std::uint64_t WriteBinary(std::uint64_t writeSize, const char* buffer, std::uint64_t offset = UINT64_MAX)
      |                          ^~~~~~~~
      |                          wint_t
AnnService/inc/Helper/DiskIO.h:217:26: error: 'uint64_t' in namespace 'std' does not name a type; did you mean 'wint_t'?
  217 |             virtual std::uint64_t ReadString(std::uint64_t& readSize, std::unique_ptr<char[]>& buffer, char delim = '\n', std::uint64_t offset = UINT64_MAX)
      |                          ^~~~~~~~
      |                          wint_t
AnnService/inc/Helper/DiskIO.h:251:26: error: 'uint64_t' in namespace 'std' does not name a type; did you mean 'wint_t'?
  251 |             virtual std::uint64_t WriteString(const char* buffer, std::uint64_t offset = UINT64_MAX)
      |                          ^~~~~~~~
      |                          wint_t
AnnService/inc/Helper/DiskIO.h:256:26: error: 'uint64_t' in namespace 'std' does not name a type; did you mean 'wint_t'?
  256 |             virtual std::uint64_t TellP()
      |                          ^~~~~~~~
      |                          wint_t
AnnService/inc/Helper/AsyncFileReader.h: In member function 'virtual bool SPTAG::Helper::AsyncFileIO::ReadFileAsync(SPTAG::Helper::AsyncReadRequest&)':
AnnService/inc/Helper/AsyncFileReader.h:580:49: error: 'struct SPTAG::Helper::AsyncReadRequest' has no member named 'm_readSize'
  580 |                 myiocb.aio_nbytes = readRequest.m_readSize;
      |                                                 ^~~~~~~~~~
AnnService/inc/Helper/AsyncFileReader.h:581:75: error: 'struct SPTAG::Helper::AsyncReadRequest' has no member named 'm_offset'
  581 |                 myiocb.aio_offset = static_cast<std::int64_t>(readRequest.m_offset);
      |                                                                           ^~~~~~~~
AnnService/src/Helper/AsyncFileReader.cpp: In function 'void SPTAG::Helper::BatchReadFileAsync(std::vector<std::shared_ptr<DiskIO> >&, AsyncReadRequest*, int)':
AnnService/src/Helper/AsyncFileReader.cpp:73:51: error: 'struct SPTAG::Helper::AsyncReadRequest' has no member named 'm_readSize'
   73 |                 myiocb->aio_nbytes = readRequest->m_readSize;
      |                                                   ^~~~~~~~~~
AnnService/src/Helper/AsyncFileReader.cpp:74:77: error: 'struct SPTAG::Helper::AsyncReadRequest' has no member named 'm_offset'
   74 |                 myiocb->aio_offset = static_cast<std::int64_t>(readRequest->m_offset);
      |                                                                             ^~~~~~~~
make[2]: *** [AnnService/CMakeFiles/SPTAGLib.dir/build.make:233: AnnService/CMakeFiles/SPTAGLib.dir/src/Helper/AsyncFileReader.cpp.o] Error 1
make[2]: Leaving directory 'build'
make[1]: *** [CMakeFiles/Makefile2:959: AnnService/CMakeFiles/SPTAGLib.dir/all] Error 2
make[1]: Leaving directory 'build'
make: *** [Makefile:149: all] Error 2

Desktop (please complete the following information):

Additional context I verified that adding #include <cstdint> to AnnService/inc/Helper/DiskIO.h fixes the build failure, however it isn't sufficient to fix the more general problem of missing includes, the tool called include-what-you-use can help discover the missing includes.