Error when transpiling `faster-utf8-validator`

[ArniDagur]

Version of c2rust: Latest master (2e656759ee5fe) Contents of compile_commands.json:

[
    {
        "arguments": [
            "cc",
            "-c",
            "-march=native",
            "-std=c11",
            "-O2",
            "z_validate.c"
        ],
        "directory": "/sdb/src/rust-isutf8/src/zwegner",
        "file": "z_validate.c"
    }
]

Contents of z_validate.c: https://github.com/zwegner/faster-utf8-validator/blob/master/z_validate.c Output of c2rust transpile compile_commands.json:

[arni][~/…/rust-isutf8/src/zwegner][master]% c2rust transpile compile_commands.json
Transpiling z_validate.c
z_validate.c:24:10: fatal error: 'immintrin.h' file not found
#include <immintrin.h>
         ^~~~~~~~~~~~~
1 error generated.
Error while processing /sdb/src/rust-isutf8/src/zwegner/z_validate.c.
warning: Missing child 94790605216440 of node AstNode { tag: TagIfStmt, children: [Some(94790605216440), Some(94790605216376), None], loc: SrcSpan { fileid: 5, begin_line: 286, begin_column: 5, end_line: 287, end_column: 16 }, type_id: None, rvalue: LValue, macro_expansions: [], extras: [] }
Exported Clang AST was invalid. Check warnings above for unimplemented features.
--> /sdb/src/rust-isutf8/src/zwegner/z_validate.c:286:5
 [-Wclang-ast]
warning: Missing child 94790605219528 of node AstNode { tag: TagIfStmt, children: [Some(94790605219528), Some(94790605219496), None], loc: SrcSpan { fileid: 5, begin_line: 321, begin_column: 5, end_line: 332, end_column: 5 }, type_id: None, rvalue: LValue, macro_expansions: [], extras: [] }
Exported Clang AST was invalid. Check warnings above for unimplemented features.
--> /sdb/src/rust-isutf8/src/zwegner/z_validate.c:321:5
 [-Wclang-ast]
warning: Missing child 94790605219400 of node AstNode { tag: TagIfStmt, children: [Some(94790605219400), Some(94790605219384), None], loc: SrcSpan { fileid: 5, begin_line: 330, begin_column: 9, end_line: 331, end_column: 20 }, type_id: None, rvalue: LValue, macro_expansions: [], extras: [] }
Exported Clang AST was invalid. Check warnings above for unimplemented features.
--> /sdb/src/rust-isutf8/src/zwegner/z_validate.c:330:9
 [-Wclang-ast]
warning: Missing child 94790605218344 of node AstNode { tag: TagIfStmt, children: [Some(94790605218344), Some(94790605218328), None], loc: SrcSpan { fileid: 5, begin_line: 303, begin_column: 5, end_line: 318, end_column: 5 }, type_id: None, rvalue: LValue, macro_expansions: [], extras: [] }
Exported Clang AST was invalid. Check warnings above for unimplemented features.
--> /sdb/src/rust-isutf8/src/zwegner/z_validate.c:303:5
 [-Wclang-ast]
warning: Missing child 94790605215752 of node AstNode { tag: TagIfStmt, children: [Some(94790605215752), Some(94790605213856), None], loc: SrcSpan { fileid: 5, begin_line: 277, begin_column: 5, end_line: 278, end_column: 16 }, type_id: None, rvalue: LValue, macro_expansions: [], extras: [] }
Exported Clang AST was invalid. Check warnings above for unimplemented features.
--> /sdb/src/rust-isutf8/src/zwegner/z_validate.c:277:5
 [-Wclang-ast]
error: Failed to translate z_validate_utf8: bad condition type

[thedataking]

@ArniDagur Try adding -DSSE4 or -DAVX2 to your compile_commands.json. Adding either of those options lets me transpile z_validate.c without issues. Closing the issue but feel free to reopen.

As a more general comment, please make sure that you can compile the code without errors before opening issues. You can see clang emitting a fatal error because it cannot include immintrin.h so that's why you are seeing the subsequent transpiler errors because our code assumes that all includes are correctly resolved.

[rinon]

I'm not sure that was the issue. There can be some build system issues with linking against the correct version of clang.

[rinon]

Hmm, nevermind, I think @thedataking was correct, see https://github.com/zwegner/faster-utf8-validator/blob/master/rules.py#L8

[ArniDagur]

Sorry, I failed to mention that had this issue after having removed the #IFs manually.

I can compile the file just fine with both clang and gcc:

[arni][~/…/rust-isutf8/src/zwegner][master]% clang -c -DAVX2 -march=native -std=c11 -O2 z_validate.c

[arni][~/…/rust-isutf8/src/zwegner][master]% gcc -c -DAVX2 -march=native -std=c11 -O2 z_validate.c
z_validate.c:210:11: warning: missing terminating " character
  210 | #   error "No valid configuration: must define one of AVX2 or SSE4
      |           ^

but transpiling gives (original file with -DAVX2, this time):

[arni][~/…/rust-isutf8/src/zwegner][master]% c2rust transpile compile_commands.json
Transpiling z_validate.c
z_validate.c:24:10: fatal error: 'immintrin.h' file not found
#include <immintrin.h>
         ^~~~~~~~~~~~~
1 error generated.
Error while processing /sdb/src/rust-isutf8/src/zwegner/z_validate.c.
warning: Missing child 94642192901928 of node AstNode { tag: TagIfStmt, children: [Some(94642192901928), Some(94642192901864), None], loc: SrcSpan { fileid: 5, begin_line: 286, begin_column: 5, end_line: 287, end_column: 16 }, type_id: None, rvalue: LValue, macro_expansions: [], extras: [] }
Exported Clang AST was invalid. Check warnings above for unimplemented features.
--> /sdb/src/rust-isutf8/src/zwegner/z_validate.c:286:5
 [-Wclang-ast]
warning: Missing child 94642192911112 of node AstNode { tag: TagIfStmt, children: [Some(94642192911112), Some(94642192911080), None], loc: SrcSpan { fileid: 5, begin_line: 321, begin_column: 5, end_line: 332, end_column: 5 }, type_id: None, rvalue: LValue, macro_expansions: [], extras: [] }
Exported Clang AST was invalid. Check warnings above for unimplemented features.
--> /sdb/src/rust-isutf8/src/zwegner/z_validate.c:321:5
 [-Wclang-ast]
warning: Missing child 94642192909272 of node AstNode { tag: TagIfStmt, children: [Some(94642192909272), Some(94642192909256), None], loc: SrcSpan { fileid: 5, begin_line: 303, begin_column: 5, end_line: 318, end_column: 5 }, type_id: None, rvalue: LValue, macro_expansions: [], extras: [] }
Exported Clang AST was invalid. Check warnings above for unimplemented features.
--> /sdb/src/rust-isutf8/src/zwegner/z_validate.c:303:5
 [-Wclang-ast]
warning: Missing child 94642192910984 of node AstNode { tag: TagIfStmt, children: [Some(94642192910984), Some(94642192910968), None], loc: SrcSpan { fileid: 5, begin_line: 330, begin_column: 9, end_line: 331, end_column: 20 }, type_id: None, rvalue: LValue, macro_expansions: [], extras: [] }
Exported Clang AST was invalid. Check warnings above for unimplemented features.
--> /sdb/src/rust-isutf8/src/zwegner/z_validate.c:330:9
 [-Wclang-ast]
error: Failed to translate z_validate_utf8_avx2: bad condition type

after manually including the header files, it works (gives a different error):

[arni][~/…/rust-isutf8/src/zwegner][master]% c2rust transpile compile_commands.json -- -I/usr/lib/clang/9.0.0/include
Transpiling z_validate.c
error: Skipping declaration Some(Typedef { name: "__m256i_u", typ: CQualTypeId { qualifiers: Qualifiers { is_const: false, is_restrict: false, is_volatile: false }, ctype: CTypeId(9607) }, is_implicit: false }) due to error: Unsupported type Vector(CQualTypeId { qualifiers: Qualifiers { is_const: false, is_restrict: false, is_volatile: false }, ctype: CTypeId(151) }, 4)

The #include error from clang didn't ring a bell since the last two times I got this same error, it was because c2rust failed to transpile, and after updating c2rust with the fixes, it worked again. I thus had the false preconception that c2rust errors result in clang include errors. After testing, this seems not to be the case, as the following program also fails to transpile without an explicit header path given:

// test.c
#include <stdio.h>

int main() {
    printf("Hello, world!")
    return 0;
}

Transpiling test.c
In file included from test.c:2:
/usr/include/stdio.h:33:10: fatal error: 'stddef.h' file not found
#include <stddef.h>
         ^~~~~~~~~~
1 error generated.
Error while processing /sdb/src/rust-isutf8/src/zwegner/test.c.

[arni][~/…/rust-isutf8/src/zwegner][master]% c2rust transpile compile_commands.json -- -I/usr/lib/clang/9.0.0/include
Transpiling test.c

[rinon]

Looks like there are two issues here. The first is that c2rust clang can't find system headers. Then it looks like there's an issue with the type __mm256i_u. However, I suspect the second might go away if you get clang in c2rust to work.

What OS or distro is this? How did you compile c2rust? I've seen this issue when c2rust was linked against LLVM libs that didn't have the right system header path embedded.

[rinon]

@ArniDagur I've reopened this because I think this is a compiler issue, any additional info re: my questions above?

[thedataking]

@ArniDagur can you please help us with the following info so we can troubleshoot the issue:

What OS or distro is this? How did you compile c2rust? I've seen this issue when c2rust was linked against LLVM libs that didn't have the right system header path embedded.

[ArniDagur]

Shoot, I thought I had already replied. I will have access to my setup tomorrow, so I will answer then.

[ArniDagur]

I'm back to my setup now. Sorry for the delay. I've updated c2rust to 0.14.0 using the following:

Environment:

LIBCLANG_PATH=/usr/lib/llvm/9/lib64/
LLVM_CONFIG_PATH=/usr/lib/llvm/9/bin/llvm-config

Commands used:

rustup install nightly-2019-12-05
rustup component add --toolchain nightly-2019-12-05 rustfmt
cargo +nightly-2019-12-05 install --git https://github.com/immunant/c2rust.git c2rust

Clang version (clang --version):

clang version 9.0.1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm/9/bin

With the same results. My operating system is Gentoo Linux.

[rinon]

Without cargo install --force, you won't actually overwrite an existing installed c2rust, was this needed?

I don't have a gentoo install available to test quickly, but I know this has worked in the past with gentoo. There may be an issue with our cmake files and LLVM 9 on gentoo. We embed CLANG_BIN_PATH and CLANG_LIBDIR_SUFFIX into the build to get the right header paths, so if these didn't get set right, that might be the issue.

[ccope]

I'm seeing the stddef.h errors on Ubuntu 18.04 with the llvm packages installed from the Ubuntu apt repositories. I worked around it by adding "-I/usr/lib/llvm-6.0/lib/clang/6.0.0/include" to each of the commands in compile_commands.json.