clang-cl preprocessor error where VC++ works

[b1622bb8-0285-400e-ba44-ba0e5027e546]

Bugzilla Link	27380
Version	trunk
OS	Windows NT
CC	@DougGregor,@ned14

Extended Description

The following code compiles without error using VC++ 14 but gives errors using the latest clang-cl from the 'trunk':

#define TEST_MACRO(x,y) TEST_EXPAND(2, 0, TEST_SOMETHING(x,y,1))
#define TEST_SOMETHING(x,y,z) 2

#define TEST_EXPAND(...) TEST_EXPAND_I(TEST_OVR(TEST_NAME, __VA_ARGS__), (__VA_ARGS__))
#define TEST_OVR(name,...) TEST_SOMENAME
#define TEST_EXPAND_I(m, args) TEST_EXPAND_II(m, args)
#define TEST_EXPAND_II(m, args) TEST_CAT(m ## args,)
#define TEST_SOMENAME(x,y,z) ;

#define TEST_CAT(a, b) TEST_CAT_I(a, b)
#define TEST_CAT_I(a, b) TEST_CAT_II(~, a ## b)
#define TEST_CAT_II(p, res) res

int main()
    {
    TEST_MACRO(1,2)
    }

The clang errors are:

test_clang.cpp(17,2):  error: pasting formed 'TEST_SOMENAME(', an invalid preprocessing token [-Winvalid-token-paste]
        TEST_MACRO(1,2)
        ^
test_clang.cpp(2,25):  note: expanded from macro 'TEST_MACRO'
#define TEST_MACRO(x,y) TEST_EXPAND(2, 0, TEST_SOMETHING(x,y,1))
                        ^
test_clang.cpp(5,26):  note: expanded from macro 'TEST_EXPAND'
#define TEST_EXPAND(...) TEST_EXPAND_I(TEST_OVR(TEST_NAME, __VA_ARGS__), (__VA_ARGS__))
                         ^
test_clang.cpp(7,32):  note: expanded from macro 'TEST_EXPAND_I'
#define TEST_EXPAND_I(m, args) TEST_EXPAND_II(m, args)
                               ^
test_clang.cpp(8,44):  note: expanded from macro 'TEST_EXPAND_II'
#define TEST_EXPAND_II(m, args) TEST_CAT(m ## args,)

The clang-cl options are:

-TP /Od /Ob0 /W3 /GR /MDd  /Zc:forScope /Zc:wchar_t -fmsc-version=1900 /wd4675 /EHs -fmacro-backtrace-limit=0 -c

The VC++14 options when compiling successfully are:

/Zm800 -TP /Z7 /Od /Ob0 /W4 /GR /MDd /Zc:forScope /Zc:wchar_t /wd4675 /EHs -c

This example is a simplified version of a failure which occurs when running the Boost Preprocessor library tests with clang-cl, where the Boost Preprocessor library is setup to use the exact same workarounds for clang-cl as it does for VC++, since neither is a C++ standard conforming preprocessor.

A possible fix for this problem is not a change in the code above, even if some specific change in the code will get clang-cl to work. If clang-cl is truly emulating the non-standard VC++ preprocessor it must be able to compile preprocessor code which VC++ also compiles without errors. Thus I am reporting this problem here.

[b1622bb8-0285-400e-ba44-ba0e5027e546]

If the -Wno-invalid-token-paste option is passed to clang-cl when compiling, the code compiles without errors. The -Wno-invalid-token-paste should be the default when targeting VC++ since the VC++ preprocessor works as if -Wno-invalid-token-paste were in effect and clang targeting VC++ is emulating the VC++ preprocessor with its non-standard behavior.

[1feda113-18e2-42e3-8624-9c17d4d32ec3]

Related: An alternative to making the MSVC preprocessor emulation more like MSVC is to give MSVC ABI target users the ability to choose an ISO standard preprocessor instead whilst retaining the MSVC ABI target: https://llvm.org/bugs/show_bug.cgi?id=27169