Statis assertion failure regression on scanner.branch

[lucasoethe]

While upgrading the lexy version in Arcade to the newest version a static assertion fails that previously succeeded:

extern/lexy/include/lexy/grammar.hpp:347:36: error: static assertion failed: missing value callback overload for production
  347 |             static_assert(_detail::error<Production, Args...>,
      |                           ~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~

The upgrade was from commit 0fed7ee to the current main (d324122).

I've tried reducing the code to the minimum (non-)working example and came up with this, which does manage to reproduce the error, however it also produced an error on old lexy versions. The minimum example which works on the old version and fails on the current version contains a much larger version of AST.h and can be found here.

We figured that this is most likely related to the breaking change "Use type-erased lexy::production_info instead of Production type in lexy::parse_tree. This is technically a breaking change, as it may affect overload resolution." from release 2022.12.0 but did not manage to pin the bug down, especially since a git bisect put the regression at commit 7719894 which seems unrelated.

This seems to be somehow related to the instantiation of scanner.branch as removing the line

scanner.branch(subscripts, dsl::p<ArraySubscripts>);

causes the assertion to no longer be checked.

Do you know how this could be fixed/whether this is an issue with lexy itself?

CC'ing @MarcusVoelker

[foonathan]

Minimal repro:

#include <lexy/action/parse.hpp>
#include <lexy/callback.hpp>
#include <lexy/dsl.hpp>
#include <lexy/input/string_input.hpp>
#include <vector>

namespace dsl = lexy::dsl;

struct outer;

struct list
{
    static constexpr auto rule  = dsl::recurse<outer>;
    static constexpr auto value = lexy::callback<const char*>([](int) { return "abc"; });
};

struct parse_as
{
#if 1
    static constexpr auto rule = dsl::parse_as<const char*>(dsl::p<list>);
#else
    static constexpr auto rule = dsl::p<list>;
#endif
    static constexpr auto value = lexy::callback<int>([](const char*) { return 0; });
};

struct outer
{
    static constexpr auto rule  = dsl::p<parse_as>;
    static constexpr auto value = lexy::forward<int>;
};

int main()
{
    lexy::parse<outer>(lexy::zstring_input(""), lexy::noop);
}

fails with:

In file included from test2.cpp:1:
In file included from include/lexy/action/parse.hpp:8:
In file included from include/lexy/action/base.hpp:11:
In file included from include/lexy/dsl/base.hpp:9:
include/lexy/grammar.hpp:347:13: error: static assertion failed due to requirement '_detail::error<outer, const char *>': missing value callback overload for production
            static_assert(_detail::error<Production, Args...>,
            ^             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
include/lexy/_detail/lazy_init.hpp:130:24: note: in instantiation of function template specialization 'lexy::production_value_callback<outer>::operator()<const char *>' requested here
        return emplace(LEXY_FWD(fn)(LEXY_FWD(args)...));
                       ^
include/lexy/_detail/config.hpp:22:23: note: expanded from macro 'LEXY_FWD'
#define LEXY_FWD(...) static_cast<decltype(__VA_ARGS__)>(__VA_ARGS__)
                      ^
include/lexy/action/base.hpp:169:23: note: in instantiation of function template specialization 'lexy::_detail::lazy_init<int>::emplace_result<lexyd::_pas_handler<const char *, parse_as, lexy::_ph<lexy::_prd>>::value_callback<outer, void>, const char *>' requested here
        context.value.emplace_result(context.value_callback(), LEXY_FWD(args)...);
                      ^
include/lexy/action/base.hpp:190:34: note: in instantiation of function template specialization 'lexy::_detail::final_parser::parse<lexy::_pc<lexyd::_pas_handler<const char *, parse_as, lexy::_ph<lexy::_prd>>, void, outer, void>, lexy::_prd, const char *>' requested here
            return continuation::parse(context, reader, LEXY_FWD(args)...,
                                 ^
include/lexy/dsl/production.hpp:66:38: note: in instantiation of function template specialization 'lexy::_detail::context_finish_parser<lexy::_detail::final_parser>::parse<lexy::_pc<lexyd::_pas_handler<const char *, parse_as, lexy::_ph<lexy::_prd>>, void, outer, void>, lexy::_prd, lexy::_pc<lexyd::_pas_handler<const char *, parse_as, lexy::_ph<lexy::_prd>>, void, parse_as, void>>' requested here
                return continuation::parse(context, reader, sub_context, LEXY_FWD(args)...);
                                     ^
include/lexy/dsl/production.hpp:25:20: note: in instantiation of function template specialization 'lexyd::_prd<parse_as>::p<lexy::_detail::final_parser>::parse<lexy::_pc<lexyd::_pas_handler<const char *, parse_as, lexy::_ph<lexy::_prd>>, void, outer, void>, lexy::_prd>' requested here
    return parser::parse(context, reader);
                   ^
include/lexy/dsl/production.hpp:61:17: note: in instantiation of function template specialization 'lexyd::_parse_production<outer, lexy::_pc<lexyd::_pas_handler<const char *, parse_as, lexy::_ph<lexy::_prd>>, void, outer, void>, lexy::_prd>' requested here
            if (_parse_production<Production>(sub_context, reader))
                ^
include/lexy/dsl/production.hpp:219:63: note: in instantiation of function template specialization 'lexyd::_prd<outer>::p<lexyd::_recb<outer>::_depth_handler<lexy::_detail::final_parser>>::parse<lexy::_pc<lexyd::_pas_handler<const char *, parse_as, lexy::_ph<lexy::_prd>>, void, list, void>, lexy::_prd>' requested here
            return lexy::parser_for<_prd<Production>, depth>::parse(context, reader,
                                                              ^
include/lexy/dsl/production.hpp:25:20: note: in instantiation of function template specialization 'lexyd::_recb<outer>::p<lexy::_detail::final_parser>::parse<lexy::_pc<lexyd::_pas_handler<const char *, parse_as, lexy::_ph<lexy::_prd>>, void, list, void>, lexy::_prd>' requested here
    return parser::parse(context, reader);
                   ^
include/lexy/dsl/production.hpp:61:17: note: in instantiation of function template specialization 'lexyd::_parse_production<list, lexy::_pc<lexyd::_pas_handler<const char *, parse_as, lexy::_ph<lexy::_prd>>, void, list, void>, lexy::_prd>' requested here
            if (_parse_production<Production>(sub_context, reader))
                ^
include/lexy/dsl/parse_as.hpp:125:72: note: in instantiation of function template specialization 'lexyd::_prd<list>::p<lexy::_detail::final_parser>::parse<lexy::_pc<lexyd::_pas_handler<const char *, parse_as, lexy::_ph<lexy::_prd>>, void, parse_as, void>, lexy::_prd>' requested here
                = lexy::parser_for<Rule, lexy::_detail::final_parser>::parse(sub_context, reader);
                                                                       ^
include/lexy/dsl/production.hpp:25:20: note: in instantiation of function template specialization 'lexyd::_pas<const char *, lexyd::_prd<list>, false>::p<lexy::_detail::final_parser>::parse<lexy::_pc<lexy::_ph<lexy::_prd>, void, parse_as, void>, lexy::_prd>' requested here
    return parser::parse(context, reader);
                   ^
include/lexy/dsl/production.hpp:61:17: note: in instantiation of function template specialization 'lexyd::_parse_production<parse_as, lexy::_pc<lexy::_ph<lexy::_prd>, void, parse_as, void>, lexy::_prd>' requested here
            if (_parse_production<Production>(sub_context, reader))
                ^
include/lexy/action/base.hpp:209:32: note: in instantiation of function template specialization 'lexyd::_prd<parse_as>::p<lexy::_detail::final_parser>::parse<lexy::_pc<lexy::_ph<lexy::_prd>, void, outer, void>, lexy::_prd>' requested here
    auto rule_result = parser::parse(context, reader);
                               ^
include/lexy/action/base.hpp:229:24: note: in instantiation of function template specialization 'lexy::_do_action<lexy::_ph<lexy::_prd>, void, outer, lexy::_prd>' requested here
    auto rule_result = _do_action(context, reader);
                       ^
include/lexy/action/parse.hpp:164:22: note: in instantiation of function template specialization 'lexy::do_action<outer, lexy::parse_action<void, lexy::string_input<>, lexy::_noop>::result_type, lexy::_ph<lexy::_prd>, void, lexy::_prd>' requested here
        return lexy::do_action<Production, result_type>(handler(input_holder, sink), _state,
                     ^
include/lexy/action/parse.hpp:173:12: note: in instantiation of function template specialization 'lexy::parse_action<void, lexy::string_input<>, lexy::_noop>::operator()<outer>' requested here
    return parse_action<void, Input, ErrorCallback>(callback)(Production{}, input);
           ^
test2.cpp:35:11: note: in instantiation of function template specialization 'lexy::parse<outer, lexy::string_input<>, lexy::_noop>' requested here
    lexy::parse<outer>(lexy::zstring_input(""), lexy::noop);
          ^
1 error generated.

But succeeds if the #if 0 is changed to #if 1.

[foonathan]

I think I've fixed it. Your minimal example still doesn't compile, but I think this is because of a bug introduced during minimization. If I'm wrong, let me know.

[lucasoethe]

Thank you! Unfortunately it seems like the issue is not fully fixed, new minimum example:

#include <lexy/action/parse.hpp>
#include <lexy/callback.hpp>
#include <lexy/dsl.hpp>
#include <lexy/input/buffer.hpp>
#include <lexy_ext/report_error.hpp>
#include <vector>

namespace arcade::parser::st {
    struct Stmt{};
    using StmtList = std::vector<Stmt>;
    struct ForStmt{};
    struct CaseStmt{};
}

namespace arcade::parser::st::grammar
{

namespace dsl = lexy::dsl;

using action = lexy::parse_action<void,
                                  lexy::buffer<lexy::utf8_char_encoding>, //
                                  std::remove_const_t<decltype(lexy_ext::report_error)>>;
struct StmtList;
constexpr auto stmt_list = dsl::subgrammar<StmtList, parser::st::StmtList>;
} // namespace arcade::parser::st::grammar

LEXY_DECLARE_SUBGRAMMAR(arcade::parser::st::grammar::StmtList)

namespace arcade::parser::st::grammar
{
struct Stmt;

struct CaseStmt
{
    struct BranchStmts : lexy::scan_production<parser::st::StmtList>
    {
        template <typename Context, typename Reader>
        static scan_result scan(lexy::rule_scanner<Context, Reader> &scanner)
        {
            lexy::scan_result<parser::st::Stmt> stmt;
            scanner.branch(stmt, dsl::recurse_branch<Stmt>);

            return parser::st::StmtList{};
        }
    };

    static constexpr auto rule = dsl::p<BranchStmts>;
    static constexpr auto value = lexy::construct<parser::st::CaseStmt>;
};

struct ForStmt
{
    static constexpr auto rule = dsl::recurse<StmtList>;
    static constexpr auto value = lexy::construct<parser::st::ForStmt>;
};

struct Stmt
{

    static constexpr auto rule = dsl::p<ForStmt>;
    static constexpr auto value = lexy::construct<parser::st::Stmt>;
};

struct StmtList
{
    static constexpr auto rule = dsl::p<Stmt>;
    static constexpr auto value = lexy::construct<parser::st::Stmt>;
};
} // namespace arcade::parser::st::grammar

LEXY_DEFINE_SUBGRAMMAR(arcade::parser::st::grammar::StmtList)
LEXY_INSTANTIATE_SUBGRAMMAR(arcade::parser::st::grammar::StmtList, arcade::parser::st::grammar::action)

[foonathan]

I think you minimized the example too much - the errors isn't actually lexy fault, the callbacks are just wrong - you can't ForStmt from a StmtList but that's what you're asking lexy to do.

[lucasoethe]

Sorry this took so long, had quite a bit to do with exams. Here is a new (less) minimized version. This causes an error on the main branch, but does not error on 0fed7ee. Also for context here is an even less minimized version, which outputs the error above 11 times, but compiles fine under 0fed7ee.

[foonathan]

Minimized.

#include <lexy/action/parse.hpp>
#include <lexy/callback.hpp>
#include <lexy/dsl.hpp>
#include <lexy/input/string_input.hpp>

namespace dsl = lexy::dsl;

struct A
{};
struct B
{};

struct Pair
{
    A a;
    B b;
};

struct ProductionPair;

struct ProductionA;
LEXY_DECLARE_SUBGRAMMAR(ProductionA)

struct ProductionB : lexy::scan_production<B>
{
    template <typename Context, typename Reader>
    static scan_result scan(lexy::rule_scanner<Context, Reader>& scanner)
    {
        scanner.template parse<ProductionPair>();
        return B{};
    }
};

struct ProductionPair
{
    static constexpr auto rule  = dsl::subgrammar<ProductionA, A> + dsl::p<ProductionB>;
    static constexpr auto value = lexy::construct<Pair>;
};

int main()
{
    lexy::parse<ProductionPair>(lexy::zstring_input(""), lexy::noop);
}

[foonathan]

All your examples compile now.