microsoft / onnxruntime-extensions

onnxruntime-extensions: A specialized pre- and post- processing library for ONNX Runtime
MIT License
323 stars 84 forks source link

Unable to build the head revision #630

Closed aejazm closed 5 months ago

aejazm commented 9 months ago

Getting the following errors while building the HEAD version on Linux following instructions from here. I am attaching the full build output as well. Please let me know if I am missing something basic.

$ git clone https://github.com/microsoft/onnxruntime-extensions $ cd onnxruntime-extensions $ ./build_lib.sh --config Release --update --build --parallel .................. [ 42%] Building CXX object _deps/spm-build/src/CMakeFiles/sentencepiece-static.dir/normalizer.cc.o [ 42%] Building CXX object _deps/opencv-build/modules/core/CMakeFiles/opencv_core.dir/src/algorithm.cpp.o In file included from ./sources/onnxruntime-extensions/base/base64.cc:3: ./sources/onnxruntime-extensions/base/base64.h:8:38: error: ‘uint8_t’ was not declared in this scope 8 | bool base64_encode(const std::vector& input, std::string& encoded); | ^~~ ./sources/onnxruntime-extensions/base/base64.h:7:1: note: ‘uint8_t’ is defined in header ‘’; did you forget to ‘#include ’? 6 | #include +++ |+#include 7 | ./sources/onnxruntime-extensions/base/base64.h:8:45: error: template argument 1 is invalid 8 | bool base64_encode(const std::vector& input, std::string& encoded); | ^ ./sources/onnxruntime-extensions/base/base64.h:8:45: error: template argument 2 is invalid ./sources/onnxruntime-extensions/base/base64.h:9:60: error: ‘uint8_t’ was not declared in this scope 9 | bool base64_decode(const std::string& encoded, std::vector& raw); | ^~~ ./sources/onnxruntime-extensions/base/base64.h:9:60: note: ‘uint8_t’ is defined in header ‘’; did you forget to ‘#include ’? ./sources/onnxruntime-extensions/base/base64.h:9:67: error: template argument 1 is invalid 9 | bool base64_decode(const std::string& encoded, std::vector& raw); | ^ ./sources/onnxruntime-extensions/base/base64.h:9:67: error: template argument 2 is invalid ./sources/onnxruntime-extensions/base/base64.cc:9:38: error: ‘uint8_t’ was not declared in this scope 9 | bool base64_encode(const std::vector& input, std::string& encoded) { | ^~~ ./sources/onnxruntime-extensions/base/base64.cc:5:1: note: ‘uint8_t’ is defined in header ‘’; did you forget to ‘#include ’? 4 | #include +++ |+#include 5 | ./sources/onnxruntime-extensions/base/base64.cc:9:45: error: template argument 1 is invalid 9 | bool base64_encode(const std::vector& input, std::string& encoded) { | ^ ./sources/onnxruntime-extensions/base/base64.cc:9:45: error: template argument 2 is invalid ./sources/onnxruntime-extensions/base/base64.cc: In function ‘bool base64_encode(const int&, std::string&)’: ./sources/onnxruntime-extensions/base/base64.cc:11:27: error: request for member ‘size’ in ‘input’, which is of non-class type ‘const int’ 11 | encoded.reserve(((input.size() / 3) + (input.size() % 3 > 0)) 4); | ^~~~ ./sources/onnxruntime-extensions/base/base64.cc:11:48: error: request for member ‘size’ in ‘input’, which is of non-class type ‘const int’ 11 | encoded.reserve(((input.size() / 3) + (input.size() % 3 > 0)) 4); | ^~~~ ./sources/onnxruntime-extensions/base/base64.cc:12:3: error: ‘uint32_t’ was not declared in this scope 12 | uint32_t temp; | ^~~~ ./sources/onnxruntime-extensions/base/base64.cc:12:3: note: ‘uint32_t’ is defined in header ‘’; did you forget to ‘#include ’? ./sources/onnxruntime-extensions/base/base64.cc:13:15: error: ‘uint8_t’ was not declared in this scope 13 | std::vector::const_iterator cursor = input.begin(); | ^~~ ./sources/onnxruntime-extensions/base/base64.cc:13:15: note: ‘uint8_t’ is defined in header ‘’; did you forget to ‘#include ’? ./sources/onnxruntime-extensions/base/base64.cc:13:22: error: template argument 1 is invalid 13 | std::vector::const_iterator cursor = input.begin(); | ^ ./sources/onnxruntime-extensions/base/base64.cc:13:22: error: template argument 2 is invalid ./sources/onnxruntime-extensions/base/base64.cc:13:40: error: expected initializer before ‘cursor’ 13 | std::vector::const_iterator cursor = input.begin(); | ^~ ./sources/onnxruntime-extensions/base/base64.cc:14:36: error: request for member ‘size’ in ‘input’, which is of non-class type ‘const int’ 14 | for (size_t idx = 0; idx < input.size() / 3; idx++) { | ^~~~ ./sources/onnxruntime-extensions/base/base64.cc:15:5: error: ‘temp’ was not declared in this scope; did you mean ‘tm’? 15 | temp = (cursor++) << 16; //Convert to big endian | ^~~~ | tm ./sources/onnxruntime-extensions/base/base64.cc:15:14: error: ‘cursor’ was not declared in this scope 15 | temp = (cursor++) << 16; //Convert to big endian | ^~ ./sources/onnxruntime-extensions/base/base64.cc:23:17: error: request for member ‘size’ in ‘input’, which is of non-class type ‘const int’ 23 | switch (input.size() % 3) { | ^~~~ ./sources/onnxruntime-extensions/base/base64.cc:25:7: error: ‘temp’ was not declared in this scope; did you mean ‘tm’? 25 | temp = (cursor++) << 16; | ^~~~ | tm ./sources/onnxruntime-extensions/base/base64.cc:25:16: error: ‘cursor’ was not declared in this scope 25 | temp = (cursor++) << 16; | ^~ ./sources/onnxruntime-extensions/base/base64.cc: At global scope: ./sources/onnxruntime-extensions/base/base64.cc:43:58: error: ‘uint8_t’ was not declared in this scope 43 | bool base64_decode(const std::string& input, std::vector& decoded) { | ^~~ ./sources/onnxruntime-extensions/base/base64.cc:43:58: note: ‘uint8_t’ is defined in header ‘’; did you forget to ‘#include ’? ./sources/onnxruntime-extensions/base/base64.cc:43:65: error: template argument 1 is invalid 43 | bool base64_decode(const std::string& input, std::vector& decoded) { | ^ ./sources/onnxruntime-extensions/base/base64.cc:43:65: error: template argument 2 is invalid ./sources/onnxruntime-extensions/base/base64.cc: In function ‘bool base64_decode(const std::string&, int&)’: ./sources/onnxruntime-extensions/base/base64.cc:54:11: error: request for member ‘clear’ in ‘decoded’, which is of non-class type ‘int’ 54 | decoded.clear(); | ^~~~~ ./sources/onnxruntime-extensions/base/base64.cc:55:11: error: request for member ‘reserve’ in ‘decoded’, which is of non-class type ‘int’ 55 | decoded.reserve(((input.length() / 4) * 3) - padding); | ^~~ ./sources/onnxruntime-extensions/base/base64.cc:56:3: error: ‘uint32_t’ was not declared in this scope 56 | uint32_t temp = 0; | ^~~~ ./sources/onnxruntime-extensions/base/base64.cc:56:3: note: ‘uint32_t’ is defined in header ‘’; did you forget to ‘#include ’? ./sources/onnxruntime-extensions/base/base64.cc:61:7: error: ‘temp’ was not declared in this scope; did you mean ‘tm’? 61 | temp <<= 6; | ^~~~ | tm ./sources/onnxruntime-extensions/base/base64.cc:75:21: error: request for member ‘push_back’ in ‘decoded’, which is of non-class type ‘int’ 75 | decoded.push_back((temp >> 16) & 0x000000FF); | ^~~~~ ./sources/onnxruntime-extensions/base/base64.cc:76:21: error: request for member ‘push_back’ in ‘decoded’, which is of non-class type ‘int’ 76 | decoded.push_back((temp >> 8) & 0x000000FF); | ^~~~~ ./sources/onnxruntime-extensions/base/base64.cc:79:21: error: request for member ‘push_back’ in ‘decoded’, which is of non-class type ‘int’ 79 | decoded.push_back((temp >> 10) & 0x000000FF); | ^~~~~ ./sources/onnxruntime-extensions/base/base64.cc:88:13: error: request for member ‘push_back’ in ‘decoded’, which is of non-class type ‘int’ 88 | decoded.push_back((temp >> 16) & 0x000000FF); | ^~~~~ ./sources/onnxruntime-extensions/base/base64.cc:88:24: error: ‘temp’ was not declared in this scope; did you mean ‘tm’? 88 | decoded.push_back((temp >> 16) & 0x000000FF); | ^~~~ | tm ./sources/onnxruntime-extensions/base/base64.cc:89:13: error: request for member ‘push_back’ in ‘decoded’, which is of non-class type ‘int’ 89 | decoded.push_back((temp >> 8) & 0x000000FF); | ^~~~~ ./sources/onnxruntime-extensions/base/base64.cc:90:13: error: request for member ‘push_back’ in ‘decoded’, which is of non-class type ‘int’ 90 | decoded.push_back((temp)&0x000000FF); | ^~~~~ buildOutput.txt

wenbingl commented 8 months ago

what's the compiler that you are using, it looks uint8_t type definition is missed, which was added in C++ 11 standard. You may try to explicitly add in base64.h to see if the issue goes away

KilowattJunkie commented 8 months ago

I also ran into this issue when trying to run the library build. I'm running on a basic install of Arch Linux, using GCC 13, on the current HEAD (d47a3dd)

./build_lib.sh --config Release --update --build --parallel
/usr/bin/c++ --version
c++ (GCC) 13.2.1 20230801

I had to add <cstdint> to base64.h, and remove the std:: prefix from some of the types in segment_extraction.cc (which makes them consistent with other usages in the file), to get it compiled:

diff --git a/base/base64.h b/base/base64.h
index 121b223..e29290f 100644
--- a/base/base64.h
+++ b/base/base64.h
@@ -4,6 +4,7 @@
 #pragma once
 #include <string>
 #include <vector>
+#include <cstdint>

 bool base64_encode(const std::vector<uint8_t>& input, std::string& encoded);
 bool base64_decode(const std::string& encoded, std::vector<uint8_t>& raw);
diff --git a/operators/math/segment_extraction.cc b/operators/math/segment_extraction.cc
index 1d4871a..3d880b6 100644
--- a/operators/math/segment_extraction.cc
+++ b/operators/math/segment_extraction.cc
@@ -11,9 +11,9 @@ OrtStatusPtr segment_extraction(const ortc::Tensor<int64_t>& input,
     return OrtW::CreateStatus("[SegmentExtraction]: Expect input dimension [n] or [1,n].", ORT_INVALID_GRAPH);
   }
   const int64_t* p_data = input.Data();
-  std::vector<std::int64_t> segment_value;
-  std::vector<std::int64_t> segment_position;
-  for (std::int64_t i = 0; i < input.NumberOfElement(); i++) {
+  std::vector<int64_t> segment_value;
+  std::vector<int64_t> segment_position;
+  for (int64_t i = 0; i < input.NumberOfElement(); i++) {
     if (!p_data[i]) {
       continue;
     }

I'm not going to make a PR only because I'm not sure how the changes will affect things (especially on other platforms), but hopefully this helps if anyone else sees the same error.

leejuyuu commented 6 months ago

I am also having the same build error on Arch and GCC 13.2.1.

wenbingl commented 6 months ago

@aejazm, @KilowattJunkie, @leejuyuu , I don't have a machine with GCC-13 installed, can any of you help to validate the PR I linked here?

leejuyuu commented 6 months ago

@wenbingl I can build this branch successfully on Arch Linux, with GCC version as follows.

➜ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: /build/gcc/src/gcc/configure --enable-languages=ada,c,c++,d,fortran,go,lto,m2,objc,obj-c++ --enable-bootstrap --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://bugs.archlinux.org/ --with-build-config=bootstrap-lto --with-linker-hash-style=gnu --with-system-zlib --enable-__cxa_atexit --enable-cet=auto --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-default-ssp --enable-gnu-indirect-function --enable-gnu-unique-object --enable-libstdcxx-backtrace --enable-link-serialization=1 --enable-linker-build-id --enable-lto --enable-multilib --enable-plugin --enable-shared --enable-threads=posix --disable-libssp --disable-libstdcxx-pch --disable-werror
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 13.2.1 20230801 (GCC) 

The tests passed, but I see some warnings

Click me ``` ➜ ./bin/extensions_test [==========] Running 24 tests from 8 test suites. [----------] Global test environment set-up. [----------] 2 tests from Exceptions [ RUN ] Exceptions.TestApiTryCatch_ThrowInModelLoad 2024-03-14 11:01:24.359698060 [E:onnxruntime:, inference_session.cc:1980 operator()] Exception during initialization: 1: Throw in ctor [ OK ] Exceptions.TestApiTryCatch_ThrowInModelLoad (24 ms) [ RUN ] Exceptions.TestApiTryCatch_ThrowInModelExecution 2024-03-14 11:01:24.362290501 [E:onnxruntime:, sequential_executor.cc:514 ExecuteKernel] Non-zero status code returned while running ExceptionalCustomOp2 node. Name:'custom' Status Message: 1: Throw in Compute [ OK ] Exceptions.TestApiTryCatch_ThrowInModelExecution (3 ms) [----------] 2 tests from Exceptions (28 ms total) [----------] 6 tests from utils [ RUN ] utils.test_ort_case [ OK ] utils.test_ort_case (3 ms) [ RUN ] utils.test_get_str_attr [ OK ] utils.test_get_str_attr (3 ms) [ RUN ] utils.test_string_join [ OK ] utils.test_string_join (2 ms) [ RUN ] utils.test_string_join_values_empty_string [ OK ] utils.test_string_join_values_empty_string (3 ms) [ RUN ] utils.test_string_join_dims_zero_values_empty [ OK ] utils.test_string_join_dims_zero_values_empty (1 ms) [ RUN ] utils.test_string_join_dims_empty_values_scalar [ OK ] utils.test_string_join_dims_empty_values_scalar (1 ms) [----------] 6 tests from utils (16 ms total) [----------] 1 test from ustring [ RUN ] ustring.tensor_operator [ OK ] ustring.tensor_operator (0 ms) [----------] 1 test from ustring (0 ms total) [----------] 2 tests from hf_bert_tokenizer_operator [ RUN ] hf_bert_tokenizer_operator.test_default 2024-03-14 11:01:24.402566355 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,51} for output input_ids 2024-03-14 11:01:24.402598576 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,51} for output attention_mask 2024-03-14 11:01:24.402606050 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,51} for output token_type_ids 2024-03-14 11:01:24.416611605 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,57} for output input_ids 2024-03-14 11:01:24.416642223 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,57} for output attention_mask 2024-03-14 11:01:24.416650147 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,57} for output token_type_ids 2024-03-14 11:01:24.424532908 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,62} for output input_ids 2024-03-14 11:01:24.424561152 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,62} for output attention_mask 2024-03-14 11:01:24.424568846 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,62} for output token_type_ids 2024-03-14 11:01:24.439383300 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,35} for output input_ids 2024-03-14 11:01:24.439427734 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,35} for output attention_mask 2024-03-14 11:01:24.439435379 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,35} for output token_type_ids 2024-03-14 11:01:24.451172525 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,36} for output input_ids 2024-03-14 11:01:24.451207000 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,36} for output attention_mask 2024-03-14 11:01:24.451212771 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,36} for output token_type_ids 2024-03-14 11:01:24.458037512 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,37} for output input_ids 2024-03-14 11:01:24.458058993 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,37} for output attention_mask 2024-03-14 11:01:24.458066377 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,37} for output token_type_ids [ OK ] hf_bert_tokenizer_operator.test_default (79 ms) [ RUN ] hf_bert_tokenizer_operator.test_optional_output 2024-03-14 11:01:24.480062416 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,51} for output input_ids 2024-03-14 11:01:24.480086181 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,51} for output attention_mask 2024-03-14 11:01:24.492411450 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,57} for output input_ids 2024-03-14 11:01:24.492442378 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,57} for output attention_mask 2024-03-14 11:01:24.498991267 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,62} for output input_ids 2024-03-14 11:01:24.499023117 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,62} for output attention_mask 2024-03-14 11:01:24.505040392 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,35} for output input_ids 2024-03-14 11:01:24.505067473 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,35} for output attention_mask 2024-03-14 11:01:24.511076633 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,36} for output input_ids 2024-03-14 11:01:24.511107391 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,36} for output attention_mask 2024-03-14 11:01:24.518379617 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,37} for output input_ids 2024-03-14 11:01:24.518407751 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {1,53} does not match actual shape of {1,37} for output attention_mask [ OK ] hf_bert_tokenizer_operator.test_optional_output (59 ms) [----------] 2 tests from hf_bert_tokenizer_operator (139 ms total) [----------] 1 test from math_operator [ RUN ] math_operator.segment_extraction [ OK ] math_operator.segment_extraction (13 ms) [----------] 1 test from math_operator (13 ms total) [----------] 7 tests from string_operator [ RUN ] string_operator.test_string_lower [ OK ] string_operator.test_string_lower (3 ms) [ RUN ] string_operator.test_regex_split_with_offsets [ OK ] string_operator.test_regex_split_with_offsets (1 ms) [ RUN ] string_operator.test_string_ecmaregex_replace [ OK ] string_operator.test_string_ecmaregex_replace (15 ms) [ RUN ] string_operator.test_vector_to_string 2024-03-14 11:01:24.565637088 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {-1} does not match actual shape of {} for output text [ OK ] string_operator.test_vector_to_string (11 ms) [ RUN ] string_operator.test_string_to_vector [ OK ] string_operator.test_string_to_vector (10 ms) [ RUN ] string_operator.test_string_mapping 2024-03-14 11:01:24.578075401 [W:onnxruntime:, execution_frame.cc:858 VerifyOutputSizes] Expected shape from model of {} does not match actual shape of {5} for output output [ OK ] string_operator.test_string_mapping (5 ms) [ RUN ] string_operator.test_masked_fill [ OK ] string_operator.test_masked_fill (13 ms) [----------] 7 tests from string_operator (62 ms total) [----------] 4 tests from tokenizer_opertors [ RUN ] tokenizer_opertors.test_bert_tokenizer [ OK ] tokenizer_opertors.test_bert_tokenizer (91 ms) [ RUN ] tokenizer_opertors.test_bert_tokenizer_scalar [ OK ] tokenizer_opertors.test_bert_tokenizer_scalar (15 ms) [ RUN ] tokenizer_opertors.test_bert_tokenizer_decoder [ OK ] tokenizer_opertors.test_bert_tokenizer_decoder (25 ms) [ RUN ] tokenizer_opertors.test_bert_tokenizer_decoder_with_idices [ OK ] tokenizer_opertors.test_bert_tokenizer_decoder_with_idices (19 ms) [----------] 4 tests from tokenizer_opertors (151 ms total) [----------] 1 test from VisionOps [ RUN ] VisionOps.image_decode_encode [ OK ] VisionOps.image_decode_encode (145 ms) [----------] 1 test from VisionOps (145 ms total) [----------] Global test environment tear-down [==========] 24 tests from 8 test suites ran. (556 ms total) [ PASSED ] 24 tests. ```
wenbingl commented 6 months ago

Thanks for verification. The warnings are irrelevant which are caused by the glitches in the test onnx models.

leejuyuu commented 6 months ago

Got it. Thanks for fixing this!