Forward declaration errors on Clang 10

[meme]

$ clang --version
clang version 10.0.1 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
$ gcc --version
gcc (GCC) 10.1.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(Note that the following also occurs on GCC 10.)

I am not clear on why -stdlib=libc++ is explicitly specified here: https://github.com/vtil-project/VTIL-Core/blob/master/VTIL-Common/CMakeLists.txt#L37-L39. Regardless, it breaks the build and causes missing includes:

In file included from _deps/vtil-core-src/VTIL-Common/arch/arm64/arm64_assembler.cpp:39:
_deps/vtil-core-src/VTIL-Common/arch/arm64/arm64_assembler.hpp:40:10: fatal error: 'string' file not found
#include <string>

So it is necessary to remove that broken configuration from the CMakeLists.txt before you are able to reproduce the following.

---

The forward declaration of cache_value here: https://github.com/vtil-project/VTIL-Core/blob/master/VTIL-SymEx/simplifier/simplifier.cpp#L94 causes errors during template expansion for std::pair, etc. because of the declaration order here, I defined cache_map as follows:

- using cache_map = std::unordered_map<expression::reference, cache_value, signature_hasher, cache_scanner>;
+ using cache_map = std::unordered_map<expression::reference, cache_value*, signature_hasher, cache_scanner>;

(And subsequently updated all accesses of cache_value to use the arrow operator.)

Then updated the map to insert values of cache_value*

- auto [it, inserted] = map.emplace( exp, make_default<cache_value>() );
+ auto [it, inserted] = map.emplace( exp, new cache_value );

Obviously using new here is not the solution, but I am unfamiliar with the lifetimes of this value so I am wondering how you think this problem can best be solved.

I'd be happy to make a PR to fix both Clang and GCC, just wasn't able to figure out the best way to do it. As well, if you'd be open to adding the latest version of GCC and Clang to the CI I'd be happy to add that as well. Looks like it's already using Clang 10.

[can1357]

An easier way to fix it would be to change the using kv_pair decl near the line throwing the error with a dummy struct with the same alignment and casting the pointer since we don't really use the structure declaration just the offset of it in a pair.

This specific file is extremely performance critical so changing it to make double the heap allocations wouldn't be very nice.

I'll check on this later tomorrow, since I can't take an in depth look right now.

As for the cmake, I believe the CI had some issues without it since it's using outdated libs by default.

Could you try doing apt-get -yq install libc++-10-dev libc++abi-10-dev and compiling again with the -stdlib=libc++ switch? Perhaps it's caused by the stdlib you're using since the CI passes with Clang 10.

[meme]

An easier way to fix it would be to change the using kv_pair decl near the line throwing the error with a dummy struct with the same alignment and casting the pointer since we don't really use the structure declaration just the offset of it in a pair.

I see, I'll look into this and wait on you for further analysis.

As for the cmake, I believe the CI had some issues without it since it's using outdated libs by default.

Ah, might make sense if I swap that out with a version check then?

Could you try doing apt-get -yq install libc++-10-dev libc++abi-10-dev and compiling again with the -stdlib=libc++ switch? Perhaps it's caused by the stdblib you're using since the CI passes with Clang 10.

I am not using Debian, but I am using the latest available libraries here. I just checked the CI and it is very strange that they pass and my Clang does not. Regardless, I'll wait on you and we can discuss if this is something that we can easily fix anyway, or if I've got to find a new compiler :laughing:.

[can1357]

Couldn't stop myself and grabbed the laptop real quick, I've tried swapping the forward declared types, can you check if it works now?

I am not using Debian, but I am using the latest available libraries here. I just checked the CI and it is very strange that they pass and my Clang does not.

I believe your issue has to do with you using libstdc++ and not libc++ rather than a version issue but might as well fix it since it shouldn't be too big of a problem.

[meme]

Unfortunately it didn't seem to help, I've provided a build trace below (note that this is the same error I experienced earlier, too):

make

``` [ 87%] Building CXX object VTIL-SymEx/CMakeFiles/VTIL-SymEx.dir/simplifier/simplifier.cpp.o In file included from /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.cpp:28: In file included from /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.hpp:32: In file included from /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/../expressions/expression.hpp:30: In file included from /home/keegan/code/VTIL-Core/VTIL-Common/includes/vtil/utility:2: /home/keegan/code/VTIL-Core/VTIL-Common/includes/vtil/../../util/copy_on_write.hpp:169:28: warning: unknown attribute 'unlikely' ignored [-Wunknown-attributes] if ( is_temporary() ) [[unlikely]] ^ /home/keegan/code/VTIL-Core/VTIL-Common/includes/vtil/../../util/copy_on_write.hpp:180:31: warning: unknown attribute 'unlikely' ignored [-Wunknown-attributes] if ( !o.combined_value ) [[unlikely]] ^ /home/keegan/code/VTIL-Core/VTIL-Common/includes/vtil/../../util/copy_on_write.hpp:185:30: warning: unknown attribute 'unlikely' ignored [-Wunknown-attributes] if ( o.is_temporary() ) [[unlikely]] ^ /home/keegan/code/VTIL-Core/VTIL-Common/includes/vtil/../../util/copy_on_write.hpp:203:36: warning: unknown attribute 'unlikely' ignored [-Wunknown-attributes] if ( prev == o.get_entry() ) [[unlikely]] ^ /home/keegan/code/VTIL-Core/VTIL-Common/includes/vtil/../../util/copy_on_write.hpp:198:52: warning: unknown attribute 'likely' ignored [-Wunknown-attributes] else if ( combined_value && !is_temporary() ) [[likely]] ^ /home/keegan/code/VTIL-Core/VTIL-Common/includes/vtil/../../util/copy_on_write.hpp:257:28: warning: unknown attribute 'unlikely' ignored [-Wunknown-attributes] if ( is_temporary() ) [[unlikely]] ^ /home/keegan/code/VTIL-Core/VTIL-Common/includes/vtil/../../util/copy_on_write.hpp:267:30: warning: unknown attribute 'unlikely' ignored [-Wunknown-attributes] if ( dec_ref( prev ) ) [[unlikely]] ^ /home/keegan/code/VTIL-Core/VTIL-Common/includes/vtil/../../util/copy_on_write.hpp:263:46: warning: unknown attribute 'likely' ignored [-Wunknown-attributes] else if ( get_ref( get_entry() ) != 1 ) [[likely]] ^ /home/keegan/code/VTIL-Core/VTIL-Common/includes/vtil/../../util/copy_on_write.hpp:308:28: warning: unknown attribute 'unlikely' ignored [-Wunknown-attributes] if ( is_temporary() ) [[unlikely]] ^ In file included from /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.cpp:28: In file included from /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.hpp:29: In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/iterator:66: In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/streambuf_iterator.h:35: In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/streambuf:41: In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/ios_base.h:41: In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/locale_classes.h:40: In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/string:40: In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/char_traits.h:39: In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/stl_algobase.h:64: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/stl_pair.h:218:11: error: field has incomplete type 'vtil::symbolic::simplifier_state::cache_value' _T2 second; ///< The second member ^ /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/ext/aligned_buffer.h:91:28: note: in instantiation of template class 'std::pair' requested here : std::aligned_storage ^ /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/hashtable_policy.h:233:43: note: in instantiation of template class '__gnu_cxx::__aligned_buffer >' requested here __gnu_cxx::__aligned_buffer<_Value> _M_storage; ^ /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/hashtable_policy.h:279:40: note: in instantiation of template class 'std::__detail::_Hash_node_value_base >' requested here struct _Hash_node<_Value, false> : _Hash_node_value_base<_Value> ^ /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/hashtable_policy.h:1974:25: note: in instantiation of template class 'std::__detail::_Hash_node, false>' requested here rebind_traits; ^ /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/hashtable.h:184:15: note: in instantiation of template class 'std::__detail::_Hashtable_alloc, false> > >' requested here private __detail::_Hashtable_alloc< ^ /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/unordered_map.h:105:18: note: in instantiation of template class 'std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >' requested here _Hashtable _M_h; ^ /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.cpp:115:4: note: in instantiation of template class 'std::unordered_map, std::equal_to, std::allocator > >' requested here cache_map_pdecl::const_iterator iterator = {}; ^ /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.cpp:94:10: note: definition of 'vtil::symbolic::simplifier_state::cache_value' is not complete until the closing '}' struct cache_value ^ In file included from /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.cpp:28: In file included from /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.hpp:30: In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/unordered_map:46: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/hashtable.h:218:24: error: type 'std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::__value_alloc_traits' (aka 'int') cannot be used prior to '::' because it has no members typedef typename __value_alloc_traits::pointer pointer; ^ /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/unordered_map.h:105:18: note: in instantiation of template class 'std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >' requested here _Hashtable _M_h; ^ /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.cpp:115:4: note: in instantiation of template class 'std::unordered_map, std::equal_to, std::allocator > >' requested here cache_map_pdecl::const_iterator iterator = {}; ^ In file included from /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.cpp:28: In file included from /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.hpp:30: In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/unordered_map:46: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/hashtable.h:219:24: error: type 'std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::__value_alloc_traits' (aka 'int') cannot be used prior to '::' because it has no members typedef typename __value_alloc_traits::const_pointer const_pointer; ^ In file included from /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.cpp:28: In file included from /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.hpp:30: In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/unordered_map:47: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/unordered_map.h:113:36: error: no type named 'mapped_type' in 'std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >' typedef typename _Hashtable::mapped_type mapped_type; ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~ /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.cpp:115:4: note: in instantiation of template class 'std::unordered_map, std::equal_to, std::allocator > >' requested here cache_map_pdecl::const_iterator iterator = {}; ^ In file included from /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.cpp:28: In file included from /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.hpp:30: In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/unordered_map:47: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/unordered_map.h:114:36: error: no type named 'hasher' in 'std::_Hashtable, std::allocator >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >' typedef typename _Hashtable::hasher hasher; ~~~~~~~~~~~~~~~~~~~~~^~~~~~ In file included from /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.cpp:28: In file included from /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.hpp:30: In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/unordered_map:46: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/hashtable.h:218:24: error: type 'std::_Hashtable, std::allocator >, std::__detail::_Select1st, vtil::symbolic::simplifier_state::cache_scanner, vtil::symbolic::simplifier_state::signature_hasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::__value_alloc_traits' (aka 'int') cannot be used prior to '::' because it has no members typedef typename __value_alloc_traits::pointer pointer; ^ /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/unordered_map.h:105:18: note: in instantiation of template class 'std::_Hashtable, std::allocator >, std::__detail::_Select1st, vtil::symbolic::simplifier_state::cache_scanner, vtil::symbolic::simplifier_state::signature_hasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >' requested here _Hashtable _M_h; ^ /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.cpp:201:13: note: in instantiation of template class 'std::unordered_map > >' requested here cache_map map{ max_cache_entries }; ^ In file included from /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.cpp:28: In file included from /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.hpp:30: In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/unordered_map:46: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/hashtable.h:219:24: error: type 'std::_Hashtable, std::allocator >, std::__detail::_Select1st, vtil::symbolic::simplifier_state::cache_scanner, vtil::symbolic::simplifier_state::signature_hasher, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::__value_alloc_traits' (aka 'int') cannot be used prior to '::' because it has no members typedef typename __value_alloc_traits::const_pointer const_pointer; ^ In file included from /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.cpp:28: In file included from /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.hpp:30: In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/unordered_map:46: In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/hashtable.h:35: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/10.1.0/../../../../include/c++/10.1.0/bits/hashtable_policy.h:349:30: error: no member named '_M_valptr' in 'std::__detail::_Hash_node, false>' { return this->_M_cur->_M_valptr(); } ~~~~~~~~~~~~ ^ /home/keegan/code/VTIL-Core/VTIL-SymEx/simplifier/simplifier.cpp:272:12: note: in instantiation of member function 'std::__detail::_Node_iterator, false, false>::operator->' requested here entry_it->second.iterator = entry_it; ^ 9 warnings and 8 errors generated. make[2]: *** [VTIL-SymEx/CMakeFiles/VTIL-SymEx.dir/build.make:160: VTIL-SymEx/CMakeFiles/VTIL-SymEx.dir/simplifier/simplifier.cpp.o] Error 1 make[1]: *** [CMakeFiles/Makefile2:647: VTIL-SymEx/CMakeFiles/VTIL-SymEx.dir/all] Error 2 make: *** [Makefile:171: all] Error 2 ```

I believe your issue has to do with you using libstdc++ and not libc++ rather than a version issue but might as well fix it since it shouldn't be too big of a problem.

This is likely it, yeah. I could install libc++, but I think if this is a (somewhat) straightforward fix, we might as well.

[meme]

https://github.com/vtil-project/VTIL-Core/commit/cd91e001213ca6f6777b17c10df8dacb4faa9720