Open cogsandsquigs opened 1 year ago
pvdrz
commented
1 year ago This sounds like C++ templates are making bindgen sad.
The part about ManuallyDrop sounds like something that we could fix. However, several issues about poorly translated generics happen because libclang doesn't have a good enough API to extract info about templates.
cogsandsquigs
commented
1 year ago Ah, ok then. I don't know how to tackle the templates issue, because I do not own or control the C++ library (obviously, I'm not tensorflow), and don't work with it too often. At least the ManuallyDrop issue can (hopefully) be fixed :p. Thanks for the input!
albx79
commented
1 year ago I'm getting exactly the same error trying to bindgen this C++ library: https://github.com/ShiqiYu/libfacedetection
I'm running this command:
bindgen facedetectcnn.h --allowlist-type=CDataBlob --allowlist-type=FaceRect --allowlist-type=ConvInfoStruct --allowlist-type=Filters --allowlist-function=objectdetect_cnn --default-non-copy-union-style=manually_drop -- -x c++ > libfacedetect.rsand then compiling with rustc libfacedetect.rs --crate-type lib generates the following error:
error[E0740]: field must implement `Copy` or be wrapped in `ManuallyDrop<...>` to be used in a union
--> libfacedetect.rs:264:5
|
264 | pub _M_val: _Tp,
| ^^^^^^^^^^^^^^^
|
= note: union fields must not have drop side-effects, which is currently enforced via either `Copy` or `ManuallyDrop<...>`
help: wrap the field type in `ManuallyDrop<...>`
|
264 | pub _M_val: std::mem::ManuallyDrop<_Tp>,
| +++++++++++++++++++++++ +I'm attaching the required header files, so you don't have to go thru libfacedetection build process if you want to reproduce the error.
mfiumara
commented
5 months ago Background: I am currently trying to build tflite-micro bindings for rust, as the previous ones are very out of date (2-3 years out of date!). I am pulling from this repo, and my build script (if you wanna look at it, it's rather expansive unfortunately) is here. A lot of the code was copied from the original
tfmicrocrate, which is now out of date :(. The process looks something like this:1. I run [this python file](https://github.com/tensorflow/tflite-micro/blob/c8079bb67847c610eabf0496e4da07224fea02e6/tensorflow/lite/micro/tools/project_generation/create_tflm_tree.py) to build a source in a temp output in `target` to bind to. This is necessary because for some reason, tflite won't compile properly otherwise (I have friends who have this same issue when building for c/c++ directly). 2. I bind `libm` as a dependency, which has no issues. 3. I create bindings with `bindgen`, which succeeds yet produces compilation errors down the line. 4. I compile inline c++ for the library in `src` (currently not in use right now as I'm just trying to get bindings to work). 5. I compile the library itself successfully. 6. `rustc` compiles my library, and errors out on the buggy bindings (see below).I've tried compiling older versions, but always get the same error, so this leads me to believe that this is a
bindgen-based issue and not atfliteone (as they have CI/CD that reports successful compilations/tests). I've also done a cursory search at past issues here and at the tensorflow repo, but nothing brought up is about my issue :((.I'm sorry that this is such a large amount of code you're recieving, but I really appreciate your response (even if it is just "oh look you forgot a setting in the docs, its right here" -- at least that would get the job done!).
Here are the requested files below:
Input C/C++ Header
Header
__bindgen.cppfile. Note that it links into thetargetdirectory, which is where the processed TFLM out put lives to compile from.The entire header is as thus:
#include "/home/ianp/Projects/tflite-micro/target/debug/build/tflite_micro-0cda1779f845e9e1/out/tflite-micro/tensorflow/lite/micro/kernels/micro_ops.h" #include "/home/ianp/Projects/tflite-micro/target/debug/build/tflite_micro-0cda1779f845e9e1/out/tflite-micro/tensorflow/lite/micro/micro_op_resolver.h" #include "/home/ianp/Projects/tflite-micro/target/debug/build/tflite_micro-0cda1779f845e9e1/out/tflite-micro/tensorflow/lite/micro/tflite_bridge/micro_error_reporter.h" #include "/home/ianp/Projects/tflite-micro/target/debug/build/tflite_micro-0cda1779f845e9e1/out/tflite-micro/tensorflow/lite/micro/micro_interpreter.h"However, I was able to reduce it down to one line that errors out (by manually knocking out headers in the bindgen config):
#include "/home/ianp/Projects/tflite-micro/target/debug/build/tflite_micro-0cda1779f845e9e1/out/tflite-micro/tensorflow/lite/micro/micro_op_resolver.h"
__bindgen.iiis not provided as of right now because it is so large that github lags when I type if I have it here! EDIT: I just realized I can use pastebin, but it goes over the maximum of 512 kilobytes.Bindgen Invocation
Invocation
let bindings = bindgen_cross_builder() // Configuration .allowlist_recursively(true) // Allow types under types. .prepend_enum_name(false) // Don't preprend the enum name to variants -- it's ugly! .impl_debug(true) // Let us debug the types. .with_codegen_config(CodegenConfig::TYPES) // TODO: figure out what this does. .layout_tests(false) .enable_cxx_namespaces() .derive_copy(true) .derive_default(true) .derive_partialeq(true) .derive_eq(true) .size_t_is_usize(true) .generate_inline_functions(true) // Generate inline functions. .use_core() // Use core instead of std. .ctypes_prefix("cty") // Use cty instead of std. .default_non_copy_union_style(NonCopyUnionStyle::ManuallyDrop) // Fix issue where `ManuallyDrop` is not wrapping values. .default_enum_style(EnumVariation::Rust { non_exhaustive: false, }) // Types .allowlist_type("tflite::MicroErrorReporter") .opaque_type("tflite::MicroErrorReporter") .allowlist_type("tflite::Model") .opaque_type("tflite::Model") .allowlist_type("tflite::MicroInterpreter") .opaque_type("tflite::MicroInterpreter") .allowlist_type("tflite::ops::micro::AllOpsResolver") .opaque_type("tflite::ops::micro::AllOpsResolver") .allowlist_type("TfLiteTensor") .allowlist_type("FrontendState") .allowlist_type("FrontendConfig") .allowlist_type("FrontendOutput") // Types - blocklist .blocklist_type("std") .blocklist_type("tflite::Interpreter_TfLiteDelegatePtr") .blocklist_type("tflite::Interpreter_State") // Headers .header(format!( "{}/tensorflow/lite/micro/kernels/micro_ops.h", tensorflow_location.to_string_lossy() )) .header(format!( "{}/tensorflow/lite/micro/micro_op_resolver.h", tensorflow_location.to_string_lossy() )) .header(format!( "{}/tensorflow/lite/micro/tflite_bridge/micro_error_reporter.h", tensorflow_location.to_string_lossy() )) .header(format!( "{}/tensorflow/lite/micro/micro_interpreter.h", tensorflow_location.to_string_lossy() )) // Inclusions .clang_arg(format!( "-include{}/tensorflow/lite/micro/micro_common.h", tensorflow_location.to_string_lossy() )) .clang_arg(format!("-I{}", tensorflow_location.to_string_lossy())) .clang_arg(format!( "-I{}/tensorflow", tensorflow_location.to_string_lossy() )) .clang_arg(format!( "-I{}/third_party/flatbuffers/include", tensorflow_location.to_string_lossy() )) // -> flatbuffers/flatbuffers.h .clang_arg(format!( "-I{}/tensorflow/lite", tensorflow_location.to_string_lossy() )) .clang_arg(format!( "-I{}/tensorflow/lite/micro", tensorflow_location.to_string_lossy() )) // -> micro/micro_common.h .clang_arg(format!( "-I{}/tensorflow/lite/c", tensorflow_location.to_string_lossy() )) // -> c/common.h // Others .clang_arg("-fretain-comments-from-system-headers") // Allow for parsing comments to create docs. .clang_arg("-DGEMMLOWP_ALLOW_SLOW_SCALAR_FALLBACK") .clang_arg("-xc++") .clang_arg("-std=c++17"); // C++17 is required for flatbuffersInvocation calls
bindgen_cross_builder, which is detailed in my repository inbuild/bind.rs.Actual Results
Output of
cargo build -vvBecause the generated types (even with reduced
#includes) are so big, I've linked the reduced version here as a pastebin. The lines of interest in the above (reduced) file are lines 1383, 1520 and 1701.Expected Results
First, I would expect that
_M_valwould be wrapped instd::mem::ManuallyDrop, as I have setdefault_non_copy_union_style(NonCopyUnionStyle::ManuallyDrop). Yet, that union field is not wrapped inManuallyDrop!I have no idea what is going on with
_Rb_tree_insert_return_type, but I'm also pretty sure that's anotherbindgenerror, asbindgenseems to be getting confused about the types of some inputs and outputs, and forgetting to place generics and/or invoke native types somewhere.Again, thanks for your time looking over my error. I appreciate both your efforts and this tool (
bindgenis kinda amazing tbh). I hope I've reduced the burden of searching for the bug somewhat, because that's a rather big library (I could probably have done more, but it's getting late where I am).
Diid you manage to get this to work? I'm trying to do exactly what you are describing here but hitting the same roadblock.
I managed to get CI passing for tfmicro for the stm32 target by the way, have you tried cross-compiling? (https://github.com/mfiumara/tfmicro/actions/runs/8530357758/job/23367982419)
Background: I am currently trying to build tflite-micro bindings for rust, as the previous ones are very out of date (2-3 years out of date!). I am pulling from this repo, and my build script (if you wanna look at it, it's rather expansive unfortunately) is here. A lot of the code was copied from the original
tfmicrocrate, which is now out of date :(. The process looks something like this:targetto bind to. This is necessary because for some reason, tflite won't compile properly otherwise (I have friends who have this same issue when building for c/c++ directly).libmas a dependency, which has no issues.bindgen, which succeeds yet produces compilation errors down the line.src(currently not in use right now as I'm just trying to get bindings to work).rustccompiles my library, and errors out on the buggy bindings (see below).I've tried compiling older versions, but always get the same error, so this leads me to believe that this is a
bindgen-based issue and not atfliteone (as they have CI/CD that reports successful compilations/tests). I've also done a cursory search at past issues here and at the tensorflow repo, but nothing brought up is about my issue :((.I'm sorry that this is such a large amount of code you're recieving, but I really appreciate your response (even if it is just "oh look you forgot a setting in the docs, its right here" -- at least that would get the job done!).
Here are the requested files below:
Input C/C++ Header
Header `__bindgen.cpp` file. Note that it links into the `target` directory, which is where the processed TFLM out put lives to compile from.
The entire header is as thus: ```C++ #include "/home/ianp/Projects/tflite-micro/target/debug/build/tflite_micro-0cda1779f845e9e1/out/tflite-micro/tensorflow/lite/micro/kernels/micro_ops.h" #include "/home/ianp/Projects/tflite-micro/target/debug/build/tflite_micro-0cda1779f845e9e1/out/tflite-micro/tensorflow/lite/micro/micro_op_resolver.h" #include "/home/ianp/Projects/tflite-micro/target/debug/build/tflite_micro-0cda1779f845e9e1/out/tflite-micro/tensorflow/lite/micro/tflite_bridge/micro_error_reporter.h" #include "/home/ianp/Projects/tflite-micro/target/debug/build/tflite_micro-0cda1779f845e9e1/out/tflite-micro/tensorflow/lite/micro/micro_interpreter.h" ``` However, I was able to reduce it down to one line that errors out (by manually knocking out headers in the bindgen config): ```C++ #include "/home/ianp/Projects/tflite-micro/target/debug/build/tflite_micro-0cda1779f845e9e1/out/tflite-micro/tensorflow/lite/micro/micro_op_resolver.h" ```__bindgen.iiis not provided as of right now because it is so large that github lags when I type if I have it here! EDIT: I just realized I can use pastebin, but it goes over the maximum of 512 kilobytes.Bindgen Invocation
Invocation
```Rust let bindings = bindgen_cross_builder() // Configuration .allowlist_recursively(true) // Allow types under types. .prepend_enum_name(false) // Don't preprend the enum name to variants -- it's ugly! .impl_debug(true) // Let us debug the types. .with_codegen_config(CodegenConfig::TYPES) // TODO: figure out what this does. .layout_tests(false) .enable_cxx_namespaces() .derive_copy(true) .derive_default(true) .derive_partialeq(true) .derive_eq(true) .size_t_is_usize(true) .generate_inline_functions(true) // Generate inline functions. .use_core() // Use core instead of std. .ctypes_prefix("cty") // Use cty instead of std. .default_non_copy_union_style(NonCopyUnionStyle::ManuallyDrop) // Fix issue where `ManuallyDrop` is not wrapping values. .default_enum_style(EnumVariation::Rust { non_exhaustive: false, }) // Types .allowlist_type("tflite::MicroErrorReporter") .opaque_type("tflite::MicroErrorReporter") .allowlist_type("tflite::Model") .opaque_type("tflite::Model") .allowlist_type("tflite::MicroInterpreter") .opaque_type("tflite::MicroInterpreter") .allowlist_type("tflite::ops::micro::AllOpsResolver") .opaque_type("tflite::ops::micro::AllOpsResolver") .allowlist_type("TfLiteTensor") .allowlist_type("FrontendState") .allowlist_type("FrontendConfig") .allowlist_type("FrontendOutput") // Types - blocklist .blocklist_type("std") .blocklist_type("tflite::Interpreter_TfLiteDelegatePtr") .blocklist_type("tflite::Interpreter_State") // Headers .header(format!( "{}/tensorflow/lite/micro/kernels/micro_ops.h", tensorflow_location.to_string_lossy() )) .header(format!( "{}/tensorflow/lite/micro/micro_op_resolver.h", tensorflow_location.to_string_lossy() )) .header(format!( "{}/tensorflow/lite/micro/tflite_bridge/micro_error_reporter.h", tensorflow_location.to_string_lossy() )) .header(format!( "{}/tensorflow/lite/micro/micro_interpreter.h", tensorflow_location.to_string_lossy() )) // Inclusions .clang_arg(format!( "-include{}/tensorflow/lite/micro/micro_common.h", tensorflow_location.to_string_lossy() )) .clang_arg(format!("-I{}", tensorflow_location.to_string_lossy())) .clang_arg(format!( "-I{}/tensorflow", tensorflow_location.to_string_lossy() )) .clang_arg(format!( "-I{}/third_party/flatbuffers/include", tensorflow_location.to_string_lossy() )) // -> flatbuffers/flatbuffers.h .clang_arg(format!( "-I{}/tensorflow/lite", tensorflow_location.to_string_lossy() )) .clang_arg(format!( "-I{}/tensorflow/lite/micro", tensorflow_location.to_string_lossy() )) // -> micro/micro_common.h .clang_arg(format!( "-I{}/tensorflow/lite/c", tensorflow_location.to_string_lossy() )) // -> c/common.h // Others .clang_arg("-fretain-comments-from-system-headers") // Allow for parsing comments to create docs. .clang_arg("-DGEMMLOWP_ALLOW_SLOW_SCALAR_FALLBACK") .clang_arg("-xc++") .clang_arg("-std=c++17"); // C++17 is required for flatbuffers ```Invocation calls
bindgen_cross_builder, which is detailed in my repository inbuild/bind.rs.Actual Results
Output of `cargo build -vv`
```console <... previous (non-erroring) compilation of c/c++, several hundred (thousand?) lines...> [tflite_micro 0.1.0] Building TFLM from source took 38.255181419s Running `CARGO=/home/ianp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo CARGO_CRATE_NAME=tflite_micro CARGO_MANIFEST_DIR=/home/ianp/Projects/tflite-micro CARGO_PKG_AUTHORS='Ian PrattBecause the generated types (even with reduced
#includes) are so big, I've linked the reduced version here as a pastebin. The lines of interest in the above (reduced) file are lines 1383, 1520 and 1701.Expected Results
First, I would expect that
_M_valwould be wrapped instd::mem::ManuallyDrop, as I have setdefault_non_copy_union_style(NonCopyUnionStyle::ManuallyDrop). Yet, that union field is not wrapped inManuallyDrop!I have no idea what is going on with
_Rb_tree_insert_return_type, but I'm also pretty sure that's anotherbindgenerror, asbindgenseems to be getting confused about the types of some inputs and outputs, and forgetting to place generics and/or invoke native types somewhere.Again, thanks for your time looking over my error. I appreciate both your efforts and this tool (
bindgenis kinda amazing tbh). I hope I've reduced the burden of searching for the bug somewhat, because that's a rather big library (I could probably have done more, but it's getting late where I am).