llvm / llvm-project

The LLVM Project is a collection of modular and reusable compiler and toolchain technologies.
http://llvm.org
Other
28.06k stars 11.59k forks source link

C++ modules appear to be exceedignly strict with intrinsic headers #98021

Open chriselrod opened 2 months ago

chriselrod commented 2 months ago

That is, code using immintrin.h tends to fail to compile when using modules while working fine with headers. I'll try to produce a minimal example in the next few hours. For now, I have an example using boost_unordered. When problems showed up in my own code using intrinsics, I could generally fix it by declaring all arguments as variables, and then passing the lvalues to the intriinsic function.

Hello.cxxm:

#ifndef USE_HEADERS
module;
#endif
#include <boost/unordered/unordered_flat_map.hpp>
#include <iostream>

#ifndef USE_HEADERS
export module Hello;
export {
#endif
  void hello() { std::cout << "Hello World!\n"; }
  template <typename K, typename V> using map = boost::unordered_flat_map<K, V>;
#ifndef USE_HEADERS
}
#endif

user.cpp:

#ifndef USE_HEADERS
import Hello;
#else
#include "hello.cxxm"
#endif

int main() {
  hello();
  int x = 0;
  long y = 0;
  map<int*,long*> m;
  m[&x] = &y;
  [[maybe_unused]] auto f = m.find(&x);
  return 0;
}

Compiling with headers:

$ clang++ -std=c++23 use.cpp -DUSE_HEADERS -o Hello.out
$ ./Hello.out
Hello World!

With modules:

$ clang++ -std=c++23 --precompile hello.cxxm -o M-hello.pcm
$ clang++ -std=c++23 use.cpp -fmodule-file=Hello=M-hello.pcm M-hello.pcm -o Hello_mod.out

results in

/usr/include/boost/unordered/detail/foa/core.hpp:293:7: error: no matching function for call to '_mm_cmpeq_epi8'
  293 |       _mm_cmpeq_epi8(load_metadata(),_mm_setzero_si128()))&0x7FFF;
      |       ^~~~~~~~~~~~~~
/usr/include/boost/unordered/detail/foa/core.hpp:309:14: note: in instantiation of member function 'boost::unordered::detail::foa::group15<boost::unordered::detail::foa::plain_integral>::match_available' requested here
  309 |     return (~match_available())&0x7FFF;

I could file this as a boost_unordered issue or make a PR there, as I've generally found I can work around the problem. But I'll see about creating a minimal reproducer using #include <immintrin.h> directlry that works with headers but fails with modules.

llvmbot commented 2 months ago

@llvm/issue-subscribers-clang-modules

Author: Chris Elrod (chriselrod)

That is, code using `immintrin.h` tends to fail to compile when using modules while working fine with headers. I'll try to produce a minimal example in the next few hours. For now, I have an example using boost_unordered. When problems showed up in my own code using intrinsics, I could generally fix it by declaring all arguments as variables, and then passing the lvalues to the intriinsic function. Hello.cxxm: ```c++ #ifndef USE_HEADERS module; #endif #include <boost/unordered/unordered_flat_map.hpp> #include <iostream> #ifndef USE_HEADERS export module Hello; export { #endif void hello() { std::cout << "Hello World!\n"; } template <typename K, typename V> using map = boost::unordered_flat_map<K, V>; #ifndef USE_HEADERS } #endif ``` user.cpp: ```c++ #ifndef USE_HEADERS import Hello; #else #include "hello.cxxm" #endif int main() { hello(); int x = 0; long y = 0; map<int*,long*> m; m[&x] = &y; [[maybe_unused]] auto f = m.find(&x); return 0; } ``` Compiling with headers: ```sh $ clang++ -std=c++23 use.cpp -DUSE_HEADERS -o Hello.out $ ./Hello.out Hello World! ``` With modules: ```sh $ clang++ -std=c++23 --precompile hello.cxxm -o M-hello.pcm $ clang++ -std=c++23 use.cpp -fmodule-file=Hello=M-hello.pcm M-hello.pcm -o Hello_mod.out ``` results in ``` /usr/include/boost/unordered/detail/foa/core.hpp:293:7: error: no matching function for call to '_mm_cmpeq_epi8' 293 | _mm_cmpeq_epi8(load_metadata(),_mm_setzero_si128()))&0x7FFF; | ^~~~~~~~~~~~~~ /usr/include/boost/unordered/detail/foa/core.hpp:309:14: note: in instantiation of member function 'boost::unordered::detail::foa::group15<boost::unordered::detail::foa::plain_integral>::match_available' requested here 309 | return (~match_available())&0x7FFF; ``` I could file this as a `boost_unordered` issue or make a PR there, as I've generally found I can work around the problem. But I'll see about creating a minimal reproducer using `#include <immintrin.h>` directlry that works with headers but fails with modules.
chriselrod commented 2 months ago

Here is the minimal example using immintrin.h:

hello.cxxm:

#ifndef USE_HEADERS
module;
#endif

#include <concepts>
#include <immintrin.h>
#include <iostream>

#ifndef USE_HEADERS
export module Hello;
export {
#endif
  void hello() { std::cout << "Hello World!\n"; }
  template <typename T> auto vload128(const T *p) {
    if constexpr (std::same_as<T, float>) {
      return _mm_loadu_ps(p);
    } else if constexpr (std::same_as<T, double>) {
      return _mm_loadu_pd(p);
    } else {
      return _mm_loadu_si128(reinterpret_cast<const __m128i *>(p));
    }
  }
#ifndef USE_HEADERS
}
#endif

use.cpp:

#ifndef USE_HEADERS
import Hello;
#else
#include "hello.cxxm"
#endif

int main() {
  hello();
  float x[4];
  [[maybe_unused]] auto v = vload128(x);
  return 0;
}

Compiling with headers:

$ clang++ -std=c++23 use.cpp -DUSE_HEADERS -o Hello.
out
$ ./Hello.out
Hello World!

Compiling with modules:

$ clang++ -std=c++23 --precompile hello.cxxm -o M-he
llo.pcm
$ clang++ -std=c++23 use.cpp -fmodule-file=Hello=M-hell
o.pcm M-hello.pcm -o Hello_mod.out
In file included from use.cpp:2:
/home/chriselrod/Documents/progwork/cxx/experiments/modules/hello.cxxm:16:14: error: no matching function for call to '_mm_loadu_ps'
   16 |       return _mm_loadu_ps(p);
      |              ^~~~~~~~~~~~
use.cpp:10:29: note: in instantiation of function template specialization 'vload128<float>' requested here
   10 |   [[maybe_unused]] auto v = vload128(x);
      |                             ^
1 error generated.
chriselrod commented 2 months ago

using something like

template <typename T> auto vload128(const T *p) {
  if constexpr (std::same_as<T, float>) {
    const float *fp = p;
    return _mm_loadu_ps(fp);
  } else if constexpr (std::same_as<T, double>) {
    const double *dp = p;
    return _mm_loadu_pd(dp);
  } else {
    const __m128i *ip = reinterpret_cast<const __m128i *>(p);
    return _mm_loadu_si128(ip);
  }
}

instead allows it to compile, even though fp and dp should have the same type as p.

chriselrod commented 2 months ago

Another workaround is to declare explicit instantiations of the template within the module that defines it.

ChuanqiXu9 commented 2 months ago

I can't reproduce this in my local environment with trunk (0182f5174f7cab31f8275718ae0ac).

My standard library is libstdc++ 10.2 (this may not be relevent) in linux.

Can you try again with trunk?

jiixyj commented 2 months ago

I just stumbled across this bug as well, upgrading from a clang trunk from around end of June to this commit: https://github.com/ChuanqiXu9/clangd-for-modules/commit/e58317686e96feb5297157cc17dddd47722ec20c. I'm also using <boost/unordered/unordered_flat_map.hpp> in my code.

I managed to bisect it to 91d40ef6e369a73b0147d9153a95c3bc63e14102 . Maybe something in that commit inadvertently touched some code related to name lookup or module visibility?

It is a bit weird though that the date of that commit is after the report date of this bug...

jiixyj commented 2 months ago

I managed to work around it for now by re-applying 91d40ef6e369a73b0147d9153a95c3bc63e14102 on top of https://github.com/ChuanqiXu9/clangd-for-modules/commit/e58317686e96feb5297157cc17dddd47722ec20c . I guess reapplying it on main should also work.

jiixyj commented 2 months ago

After reapplying https://github.com/llvm/llvm-project/commit/91d40ef6e369a73b0147d9153a95c3bc63e14102 , the reproducer from here now fails again, but luckily my code doesn't seem to trigger that case.

ChuanqiXu9 commented 1 month ago

@jiixyj I've relanded https://github.com/llvm/llvm-project/pull/102287. Can you verify if this is still a problem?

jiixyj commented 1 month ago

@ChuanqiXu9 : Sadly, with 3c9e3457d7e3153f449bef047d4deb297126f446 I still get the errors when including <boost/unordered/unordered_flat_map.hpp>:

/home/jan/.conan2/p/b/boost77375cc29ad40/p/include/boost/unordered/detail/foa/core.hpp:291:12: error: no matching function for call to '_mm_movemask_epi8'
  291 |     return _mm_movemask_epi8(
      |            ^~~~~~~~~~~~~~~~~

and

/home/jan/.conan2/p/b/boost77375cc29ad40/p/include/boost/unordered/detail/foa/core.hpp:1070:36: error: no matching function for call to 'size_index_for'                                        
 1070 |     auto         groups_size_index=size_index_for<group_type,size_policy>(n);                                                                                                           
      |                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                                                                                                               

...and some more similar ones.

I looked at the differences between the original PR and the relanded version. It turns out if I make Decl::isInAnotherModuleUnit() look like the following, the errors go away:

bool Decl::isInAnotherModuleUnit() const {
  auto *M = getOwningModule();

#if 1
  if (!M || !M->isNamedModule())
    return false;
#else
  if (!M)
    return false;

  // FIXME or NOTE: maybe we need to be clear about the semantics
  // of clang header modules. e.g., if this lives in a clang header
  // module included by the current unit, should we return false
  // here?
  //
  // This is clear for header units as the specification says the
  // header units live in a synthesised translation unit. So we
  // can return false here.
  M = M->getTopLevelModule();
  if (!M->isNamedModule())
    return false;
#endif

  return M != getASTContext().getCurrentNamedModule();
}

The code between #else and #endif is the current code in the main branch.

jiixyj commented 1 month ago

This is just a shot in the dark as I'm really not familiar with this code, but maybe there should be a check for the global module again? Like this:

 bool Decl::isInAnotherModuleUnit() const {
   auto *M = getOwningModule();

   if (!M)
     return false;

+  if (M->isGlobalModule())
+    return false;
+
   // FIXME or NOTE: maybe we need to be clear about the semantics
   // of clang header modules. e.g., if this lives in a clang header
   // module included by the current unit, should we return false
   // here?
   //
   // This is clear for header units as the specification says the
   // header units live in a synthesised translation unit. So we
   // can return false here.
   M = M->getTopLevelModule();
   if (!M->isNamedModule())
     return false;

   return M != getASTContext().getCurrentNamedModule();
 }

Otherwise, getTopLevelModule() turns e.g. my.module.<global> into my.module.

Also I'm wondering if any change in isInAnotherModuleUnit() should be mirrored in isInCurrentModuleUnit()?

ChuanqiXu9 commented 1 month ago

@jiixyj Thanks for the analysis, it is helpful. But the suggested check may not be the case. Since the GMF may be in the current module unit semantically. I'll try if the minimal reproducer from @chriselrod works. If yes I guess I can fix it quickly. But if not, maybe it will be better to get a reproducer from you.

Also I'm wondering if any change in isInAnotherModuleUnit() should be mirrored in isInCurrentModuleUnit()?

Maybe not. Since a translation unit may not be a module unit. Otherwise we can implement isInAnotherModuleUnit() as !isInCurrentModuleUnit()

jiixyj commented 1 month ago

Here is a minimal reproducer. I used cvise to create this, plus some simplifications by hand.

map.cppm:

module;

static void fun(long);

template <typename = void> struct a {
  a() { fun(load()); }
  long load();
};

export module map;

export using map = a<>;

map.cpp:

import map;
map m;

Run it like:

$ clang++ -std=c++23 --precompile map.cppm -o M-map.pcm && clang++ -std=c++23 -c map.cpp -fmodule-file=map=M-map.pcm
In file included from map.cpp:1:
map.cppm:6:9: error: no matching function for call to 'fun'
    6 |   a() { fun(load()); }
      |         ^~~
map.cpp:2:5: note: in instantiation of member function 'a<>::a' requested here
    2 | map m;
      |     ^
1 error generated.
jiixyj commented 1 month ago

For the record, here is the resulting map.cppm from cvise before my manual simplifications:

```cpp module; static void _mm_movemask_epi8(long); template