Basic support for C++20 concepts

[Silveryard]

Would it be possible to add basic support for C++20 concepts? For my use case I wouldn't need any advanced details about them and it would be sufficient if the parser could detect and ignore them without generting warnings/errors.

Declaring a concept generates a warning but the parser skips over it:

#include <concepts>

class Base {};

template<typename T>
concept BaseType = std::derived_from<T, Base>;

Input flags: -v --std c++20

Output:

[preprocessor] [debug] Test.h:1: parsing include 'concepts'
[libclang parser] [debug] Test.h:3: parsing cursor of type 'ClassDecl'
[libclang parser] [debug] Test.h:6: parsing cursor of type 'UnexposedDecl'
[libclang parser] [warning] Test.h:6: unhandled cursor of kind 'UnexposedDecl'
AST for 'Test.h':
|-concepts (include directive): `#include <concepts>`
|-Base (class) [definition]: `class Base;`
+-BaseType (unexposed entity): `template<typename T>concept BaseType=std::derived_from<T,Base>;`

Using it for a templated type generates an error:

#include <concepts>

class Base {};

template<typename T>
concept BaseType = std::derived_from<T, Base>;

template<BaseType T>
class A {};

Input flags: -v --std c++20

Output:

[preprocessor] [debug] Test.h:1: parsing include 'concepts'
[libclang parser] [debug] Test.h:3: parsing cursor of type 'ClassDecl'
[libclang parser] [debug] Test.h:6: parsing cursor of type 'UnexposedDecl'
[libclang parser] [warning] Test.h:6: unhandled cursor of kind 'UnexposedDecl'
[libclang parser] [debug] Test.h:9: parsing cursor of type 'ClassTemplate'
[libclang parser] [error] Test.h:8: expected 'class', got 'BaseType'
AST for 'Test.h':
|-concepts (include directive): `#include <concepts>`
|-Base (class) [definition]: `class Base;`
+-BaseType (unexposed entity): `template<typename T>concept BaseType=std::derived_from<T,Base>;`

[foonathan]

The particular error is caused by the skip here: https://github.com/foonathan/cppast/blob/e558e2d58f519e3a83af770d460672b1d4ba2886/src/libclang/template_parser.cpp#L137. It needs to account for the presence of a concept name as well, with a potential update to cpp_template_keyword.

The following should work:

• [ ] Update https://github.com/foonathan/cppast/blob/e558e2d58f519e3a83af770d460672b1d4ba2886/include/cppast/cpp_template_parameter.hpp#L36, adding a keyword_concept.
• [ ] Updating the logic in the file linked above to detect if it's neither typename nor class, and skip over an identifier, followed by a potential balanced sequence of angle brackets to account for concepts with template parameters. skip_brackets implements a heuristic that should work https://github.com/foonathan/cppast/blob/e558e2d58f519e3a83af770d460672b1d4ba2886/src/libclang/cxtokenizer.hpp#L185
• [ ] Add a couple of test cases

If you do a PR, it'll be fixed more quickly. ;)

In the long-term, I plan on moving away from libclang to clang JSON, see #120. If I find the time, I might add proper concept support.

[Silveryard]

I implemented the proposed changes which fixes the issue described above here. This will of course only fix the errors that prevent parsing but not the warnings for the concept declaration itself.

But it will only work in this simple case. I wrote some additional tests for abbreviated function templates and requires clauses/constraints. These constraint tests are not exhaustive and do not cover conjunctions, disjunctions, atomic constraints and requires expressions.

I feel like this is not enough to warrant a PR yet. At the very least there should be some logic to ignore the different types of constraints. Skipping concept declarations to get rid of the warnings would also be nice. Abbreviated function templates are not a requirement for me but I included a test for them for completeness.

[foonathan]

Thanks for looking into it in more detail.

This will of course only fix the errors that prevent parsing but not the warnings for the concept declaration itself.

Yes, that requires support for libclang as well. I don't know the C++20 support in libclang, but it's probably bad.

I wrote some additional tests for abbreviated function templates and requires clauses/constraints. These constraint tests are not exhaustive and do not cover conjunctions, disjunctions, atomic constraints and requires expressions.

Can you post the error messages you're getting with those examples?

[Silveryard]

Can you post the error messages you're getting with those examples?

Of course, here is the output when running cppast_test with the added tests:

[libclang parser] [warning] cpp_class_template_concept.cpp:7: unhandled cursor of kind 'UnexposedDecl'
[libclang parser] [warning] cpp_class_template_templated_concept.cpp:8: unhandled cursor of kind 'UnexposedDecl'
[libclang parser] [warning] cpp_function_template_concept.cpp:7: unhandled cursor of kind 'UnexposedDecl'
[libclang parser] [warning] cpp_function_template_concept_abbreviated.cpp:8: unhandled cursor of kind 'UnexposedDecl'
[libclang parser] [error] cpp_function_template_concept_abbreviated.cpp:10: unable to find end of function prefix
[libclang parser] [error] cpp_function_template_concept_abbreviated.cpp:11: unable to find end of function prefix
[libclang parser] [error] cpp_function_template_concept_abbreviated.cpp:12: unable to find end of function prefix
[libclang parser] [error] cpp_function_template_concept_abbreviated.cpp:13: unable to find end of function prefix
[libclang parser] [error] cpp_function_template_concept_abbreviated.cpp:14: unable to find end of function prefix

cppast_test.exe is a Catch v2.13.4 host application.
Run with -? for options

-------------------------------------------------------------------------------
cpp_function_template_concept_abbreviated
-------------------------------------------------------------------------------
C:\Data\Tools\cppast\test\cpp_concept.cpp(64)
...............................................................................

C:\Data\Tools\cppast\test\test_parser.hpp(43): FAILED:
  REQUIRE( !p.error() )
with expansion:
  false

[simple file parser] [info] parsing file 'a.cpp'
[simple file parser] [info] parsing file 'b.cpp'
[simple file parser] [info] parsing file 'c.cpp'
[libclang parser] [warning] cpp_class_template_constraint.cpp:7: unhandled cursor of kind 'UnexposedDecl'
[libclang parser] [error] cpp_class_template_constraint.cpp:10: expected 'class', got 'requires'
-------------------------------------------------------------------------------
cpp_class_template_constraint
-------------------------------------------------------------------------------
C:\Data\Tools\cppast\test\cpp_constraint.cpp(9)
...............................................................................

C:\Data\Tools\cppast\test\test_parser.hpp(43): FAILED:
  REQUIRE( !p.error() )
with expansion:
  false

[libclang parser] [warning] cpp_class_template_templated_constraint.cpp:8: unhandled cursor of kind 'UnexposedDecl'
[libclang parser] [error] cpp_class_template_templated_constraint.cpp:11: expected 'class', got 'requires'
-------------------------------------------------------------------------------
cpp_class_template_templated_constraint
-------------------------------------------------------------------------------
C:\Data\Tools\cppast\test\cpp_constraint.cpp(27)
...............................................................................

C:\Data\Tools\cppast\test\test_parser.hpp(43): FAILED:
  REQUIRE( !p.error() )
with expansion:
  false

[libclang parser] [warning] cpp_function_template_constraint.cpp:7: unhandled cursor of kind 'UnexposedDecl'
[libclang parser] [warning] cpp_function_template_constraint_trailing.cpp:7: unhandled cursor of kind 'UnexposedDecl'
[libclang parser] [error] cpp_function_template_constraint_trailing.cpp:13: unable to find end of function prefix
-------------------------------------------------------------------------------
cpp_function_template_constraint_trailing
-------------------------------------------------------------------------------
C:\Data\Tools\cppast\test\cpp_constraint.cpp(64)
...............................................................................

C:\Data\Tools\cppast\test\test_parser.hpp(43): FAILED:
  REQUIRE( !p.error() )
with expansion:
  false

===============================================================================
test cases:   51 |   47 passed | 4 failed
assertions: 2361 | 2357 passed | 4 failed

[foonathan]

Hm, that's odd. The assertion is generated here, but I can't see why it would generate - you haven't touched the function prefix at all: https://github.com/foonathan/cppast/blob/main/src/libclang/function_parser.cpp#L301

Maybe the computed name or associated tokens are weird?

[Silveryard]

Fixed by https://github.com/foonathan/cppast/pull/144 :)