search

weld-project / weld

High-performance runtime for data analytics applications
https://www.weld.rs
BSD 3-Clause "New" or "Revised" License
2.99k stars 259 forks source link

"cargo test" fails with Rust 1.37.0 and LLVM/Clang 6.0.1 #473

Closed boegel closed 4 years ago

boegel commented 4 years ago

I tried installing Weld 0.3.1 on top of Rust 1.37.0 and Clang 6.0.1 (both installed from source using GCC 7.3), but ran into test failures. I ran into the same failure when using Rust 1.37.0 and Clang 8.0.1 & 9.0.0 where both were built from source using GCC 8.2.

Executed commands with output under the Details dropdown:

$ export CLANG=clang++ && cargo build --release
``` Updating crates.io index Compiling proc-macro2 v1.0.4 Compiling libc v0.2.62 Compiling unicode-xid v0.2.0 Compiling getrandom v0.1.12 Compiling cfg-if v0.1.9 Compiling syn v1.0.5 Compiling memchr v2.2.1 Compiling ppv-lite86 v0.2.5 Compiling proc-macro2 v0.4.30 Compiling winapi-build v0.1.1 Compiling lazy_static v1.4.0 Compiling unicode-xid v0.1.0 Compiling bitflags v1.1.0 Compiling autocfg v0.1.6 Compiling serde v1.0.101 Compiling ryu v1.0.0 Compiling regex v0.2.11 Compiling ucd-util v0.1.5 Compiling utf8-ranges v1.0.4 Compiling rand_core v0.4.2 Compiling log v0.4.8 Compiling winapi v0.2.8 Compiling unicode-width v0.1.6 Compiling syn v0.15.44 Compiling semver-parser v0.7.0 Compiling cbindgen v0.8.7 Compiling cc v1.0.45 Compiling remove_dir_all v0.5.2 Compiling itoa v0.4.4 Compiling ansi_term v0.11.0 Compiling strsim v0.8.0 Compiling vec_map v0.8.1 Compiling utf8-ranges v0.1.3 Compiling weld v0.3.1 (/tmp/weld-0.3.1/weld) Compiling regex-syntax v0.3.9 Compiling bitflags v0.4.0 Compiling encode_unicode v0.1.3 Compiling lazy_static v0.2.11 Compiling fnv v1.0.6 Compiling code_builder v0.1.0 Compiling thread_local v0.3.6 Compiling kernel32-sys v0.2.2 Compiling regex-syntax v0.5.6 Compiling rand_core v0.3.1 Compiling rand_jitter v0.1.4 Compiling textwrap v0.11.0 Compiling c2-chacha v0.2.2 Compiling rand_isaac v0.1.1 Compiling rand_xorshift v0.1.1 Compiling rand_hc v0.1.0 Compiling rand_chacha v0.1.1 Compiling num-traits v0.2.8 Compiling rand_pcg v0.1.2 Compiling rand v0.6.5 Compiling num-integer v0.1.41 Compiling log v0.3.9 Compiling semver v0.9.0 Compiling atty v0.2.13 Compiling memchr v0.1.11 Compiling thread-id v2.0.0 Compiling rand_os v0.1.3 Compiling time v0.1.42 Compiling nix v0.5.1 Compiling clap v2.33.0 Compiling aho-corasick v0.5.3 Compiling aho-corasick v0.6.10 Compiling thread_local v0.2.7 Compiling rand_core v0.5.1 Compiling rand_chacha v0.2.1 Compiling rand v0.7.2 Compiling quote v1.0.2 Compiling chrono v0.4.9 Compiling quote v0.6.13 Compiling regex v0.1.80 Compiling uuid v0.7.4 Compiling rustyline v1.0.0 Compiling tempfile v3.1.0 Compiling serde_derive v1.0.101 Compiling llvm-sys v60.4.1 Compiling env_logger v0.4.3 Compiling repl v0.3.0 (/tmp/weld-0.3.1/weld-repl) Compiling hdrgen v0.3.0 (/tmp/weld-0.3.1/weld-hdrgen) Compiling serde_json v1.0.40 Compiling toml v0.4.10 Compiling weld-capi v0.3.0 (/tmp/weld-0.3.1/weld-capi) Finished release [optimized] target(s) in 1m 18s ``` 
$ cargo test --release

Output for failing test:

     Running target/release/deps/complex_loop_tests-bfb702c60ba8b3fc

running 6 tests
complex_loop_tests-bfb702c60ba8b3fc: /software/Clang/6.0.1/GCC-7.3.0-2.30/llvm-6.0.1.src/lib/IR/Constants.cpp:2055: static llvm::Constant *llvm::ConstantExpr::getInsertValue(llvm::Constant *, llvm::Constant *, ArrayRef<unsigned int>, llvm::Type *): Assertion `ExtractValueInst::getIndexedType(Agg->getType(), Idxs) == Val->getType() && "insertvalue indices invalid!"' failed.
error: process didn't exit successfully: `/tmp/weld-0.3.1/target/release/deps/complex_loop_tests-bfb702c60ba8b3fc` (signal: 6, SIGABRT: process abort signal)

Full output of cargo test --release:

``` Compiling weld v0.3.1 (/tmp/weld-0.3.1/weld) Compiling hdrgen v0.3.0 (/tmp/weld-0.3.1/weld-hdrgen) Compiling repl v0.3.0 (/tmp/weld-0.3.1/weld-repl) Compiling weld-capi v0.3.0 (/tmp/weld-0.3.1/weld-capi) Finished release [optimized] target(s) in 19.41s Running target/release/deps/hdrgen-8c616f06db4a3dd8 running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Running target/release/deps/repl-f2e12a0e7733801e running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Running target/release/deps/weld-6bc0ffa5b90761a4 running 75 tests test ast::constructors::makevector_empty_test ... ok test ast::constructors::binop_test ... ok test ast::constructors::comparison_test ... ok test ast::constructors::builder_exprs_test ... ok test ast::constructors::literal_test ... ok test data::size_check ... ok test ast::hash::test_compare_different_symbols_ne ... ok test ast::hash::test_compare_different_symbols ... ok test ast::hash::test_compare_same ... ok test ast::hash::test_lambda ... ok test optimizer::transforms::cse::builder_test ... ok test ast::type_inference::infer_types_test ... ok test optimizer::transforms::cse::builder_test_3 ... ok test ast::cmp::compare_expressions ... ok test optimizer::transforms::cse::builder_test_4 ... ok test ast::uniquify::parse_and_print_uniquified_expressions ... ok test optimizer::transforms::cse::basic_test ... ok test optimizer::transforms::cse::builder_test_2 ... ok test optimizer::transforms::cse::alias_test ... ok test optimizer::transforms::cse::for_test ... ok test optimizer::transforms::cse::for_test_2 ... ok test optimizer::transforms::cse::if_test ... ok test optimizer::transforms::cse::if_test_3 ... ok test optimizer::transforms::cse::if_test_2 ... ok test optimizer::transforms::cse::if_test_4 ... ok test optimizer::transforms::cse::let_test ... ok test optimizer::transforms::cse::for_test_4 ... ok test optimizer::transforms::cse::for_test_3 ... ok test optimizer::transforms::cse::if_test_5 ... ok test optimizer::transforms::short_circuit::compound_or ... ok test optimizer::transforms::cse::if_test_6 ... ok test optimizer::transforms::short_circuit::simple_or ... ok test optimizer::transforms::short_circuit::compound_and ... ok test optimizer::transforms::cse::if_test_7 ... ok test optimizer::transforms::cse::if_test_8 ... ok test optimizer::transforms::unroller::large_merger_loop ... ok test optimizer::transforms::cse::nesting_test ... ok test optimizer::transforms::cse::many_subexprs_test ... ok test optimizer::transforms::short_circuit::complex_and_or ... ok test optimizer::transforms::short_circuit::simple_and ... ok test optimizer::transforms::short_circuit::predicated_if ... ok test optimizer::transforms::unroller::simple_appender_loop ... ok test optimizer::transforms::unroller::zipped_appender_loop ... ok test optimizer::transforms::vectorizer::predicated_appender ... ok test optimizer::transforms::vectorizer::non_vectorizable_type ... ok test optimizer::transforms::vectorizer::unpredicated_merger ... ok test optimizer::transforms::vectorizer::non_vectorizable_expr ... ok test optimizer::transforms::vectorizer::simple_merger ... ok test optimizer::transforms::vectorizer::simple_appender ... ok test optimizer::transforms::vectorizer::predicated_merger ... ok test optimizer::transforms::unroller::zipped_merger_loop ... ok test syntax::macro_processor::invalid_type_alias ... ok test syntax::macro_processor::redefinition_with_type_alias_in_type_alias ... ok test syntax::macro_processor::expr_with_type_alias ... ok test syntax::macro_processor::nested_type_alias ... ok test optimizer::transforms::inliner::inline_lets ... ok test syntax::macro_processor::invalid_type_alias_in_type_alias ... ok test syntax::macro_processor::type_alias_in_type_alias ... ok test syntax::macro_processor::basic_type_alias ... ok test syntax::macro_processor::overwrite_type_alias ... ok test syntax::macro_processor::basic_macros ... ok test syntax::parser::parse_and_print_for_expressions ... ok test syntax::parser::read_to_end_of_input ... ok test syntax::parser::parse_and_print_simple_expressions ... ok test syntax::parser::parse_and_print_literal_expressions ... ok test syntax::macro_processor::macros_introducing_symbols ... ok test syntax::parser::basic_parsing ... ok test optimizer::transforms::loop_fusion::simple_vertical_loop_fusion ... ok test optimizer::transforms::unroller::simple_merger_loop ... ok test optimizer::transforms::short_circuit::complex_and_or_2 ... ok test syntax::parser::operator_precedence ... ok test optimizer::transforms::vectorizer::zipped_input ... ok test optimizer::transforms::loop_fusion::simple_horizontal_loop_fusion ... ok test syntax::macro_processor::standard_macros ... ok test syntax::tokenizer::basic_tokenize ... ok test result: ok. 75 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Running target/release/deps/appender_tests-ccc27335d9d7c215 running 5 tests test simple_parallel_for_appender_loop ... ok test simple_for_appender_loop ... ok test large_unaryop_for_appender_loop ... ok test simple_parallel_for_multi_appender_loop ... ok test complex_parallel_for_appender_loop ... ok test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Running target/release/deps/basic_tests-d76fcc47b63a4cd6 running 25 tests test iterate_non_parallel ... ok test bool_eq ... ok test i32_cast ... ok test iterate_with_parallel_body ... ok test filter_length ... ok test f64_cast ... ok test flat_map_length ... ok test let_statement ... ok test maxmin ... ok test basic_program ... ok test map_zip_loop ... ok test if_statement ... ok test if_for_loop ... ok test program_with_args ... ok test multiple_casts ... ok test comparison ... ok test negated_arithmetic ... ok test simple_length ... ok test struct_vector_literals ... ok test negation_double ... ok test negation ... ok test reused_variable ... ok test not ... ok test serial_parlib_test ... ok test float_literals ... ok test result: ok. 25 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Running target/release/deps/binop_tests-34571d78fb7705e6 running 5 tests test struct_cmp ... ok test simple_binop ... ok test string_cmp ... ok test vector_cmp ... ok test substring_cmp ... ok test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Running target/release/deps/complex_loop_tests-bfb702c60ba8b3fc running 6 tests complex_loop_tests-bfb702c60ba8b3fc: /software/Clang/6.0.1/GCC-7.3.0-2.30/llvm-6.0.1.src/lib/IR/Constants.cpp:2055: static llvm::Constant *llvm::ConstantExpr::getInsertValue(llvm::Constant *, llvm::Constant *, ArrayRef, llvm::Type *): Assertion `ExtractValueInst::getIndexedType(Agg->getType(), Idxs) == Val->getType() && "insertvalue indices invalid!"' failed. error: process didn't exit successfully: `/tmp/weld-0.3.1/target/release/deps/complex_loop_tests-bfb702c60ba8b3fc` (signal: 6, SIGABRT: process abort signal) ```

Is this a known issue, or am I overlooking something?

Side notes:

sppalkia commented 4 years ago

Hmm, I'm not able to reproduce this, what do you see when you run llvm-config --version?

Good points re: README, I will fix those.

boegel commented 4 years ago

As expected, 6.0.1 (part of the Clang installation)

$ llvm-config --version
6.0.1
$ which llvm-config
/software/Clang/6.0.1-GCC-7.3.0-2.30/bin/llvm-config
sppalkia commented 4 years ago

Hmm okay, let me look into this; I suspect it’s because building from source enables some debug assertions or something, so that will need to be fixed.

In the meanwhile, you can try using the LLVM distribution available on brew (or follow the instructions on the README), that should work.

On Sun, Sep 22, 2019 at 1:14 PM Kenneth Hoste notifications@github.com wrote:

As expected, 6.0.1 (part of the Clang installation)

$ llvm-config --version 6.0.1 $ which llvm-config /software/Clang/6.0.1-GCC-7.3.0-2.30/bin/llvm-config

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/weld-project/weld/issues/473?email_source=notifications&email_token=AAKMEY2L3LRF6T2L53XPBJTQK7G3RA5CNFSM4IZDDQXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7JOIZA#issuecomment-533914724, or mute the thread https://github.com/notifications/unsubscribe-auth/AAKMEY2YFR74N6XAFBX2QBDQK7G3RANCNFSM4IZDDQXA .

-- Shoumik

zao commented 4 years ago

Clang/LLVM built with this build system typically has -DLLVM_ENABLE_ASSERTIONS=ON.

I've tested four different builds of the toolchain:

The distro's compiler and the 6.0.1 without assertions pass the test suite, the two builds with assertions bail out in the same manner that @boegel's did.

sppalkia commented 4 years ago

Okay, I built it from source and was able to reproduce this. Will post a patch shortly.