Completion provider (parser) doesn't respect optional prefix of same shape than subsequent mandatory parts

[sailingKieler]

Langium version: 3.0.0-next Package name: langium

The completion provider -- more precisely the completion parser -- appears to be too eager for grammar rule RHSs like

(k1=keywords '.')? k2=keywords;

with keywords: 'A' | 'B';.

In my experiments it directly matches the second call of rule keywords as soon as a A or B is given. However, some kind of local backtracking seems to be required here. Besides, with the cursor being placed directly behind a valid keyword, the completion provider doesn't propose the next possible keyword, but just the one in left of the cursor. (Actually, I run into this with k1 and k2 being assigned with cross references with different candidate scopes.)

I've prepared two tests, one with the leading part being optional as above, and one with it being mandatory. The test expectation represent the current behavior, my expected results are added in comments:

    test('Should propose all applicable keywords and respect the optional leading group', async () => {

        const grammar = `
        grammar g
        entry Main:(k1=keywords '.')? k2=keywords;
        keywords: 'A' | 'B'; 
        hidden terminal WS: /\\s+/;
        `;

        const services = await createServicesForGrammar({ grammar, module: allKeywordsCompletionProviderModule });
        const completion = expectCompletion(services);
        const text = '<|>A<|>.<|> <|>B';

        expect(
            (await parseDocument(services, text.replaceAll('<|>', ''))).parseResult.parserErrors
        ).toHaveLength(0);

        await completion({
            text,
            index: 0,
            expectedItems: ['A', 'B']
        });

        await completion({
            text,
            index: 1,
            expectedItems: ['A'] // would expect ['A', '.'] here
        });

        await completion({
            text,
            index: 2,
            expectedItems: [ ]   // would expect ['.', 'A', 'B'] here
        });

        await completion({
            text,
            index: 3,
            expectedItems: [ ]   // would expect ['A', 'B'] here
        });
    });

    const allKeywordsCompletionProviderModule = {
        lsp: {
            CompletionProvider: (services: LangiumServices) => new class extends DefaultCompletionProvider {
                protected override filterKeyword(): boolean { return true; }
            }(services)
        }
    };

    test('Should propose all applicable keywords', async () => {

        const grammar = `
        grammar g
        entry Main:(k1=keywords '.') k2=keywords;
        keywords: 'A' | 'B'; 
        hidden terminal WS: /\\s+/;
        `;

        const services = await createServicesForGrammar({ grammar, module: allKeywordsCompletionProviderModule });
        const completion = expectCompletion(services);
        const text = '<|>A<|>.<|> <|>B';

        expect(
            (await parseDocument(services, text.replaceAll('<|>', ''))).parseResult.parserErrors
        ).toHaveLength(0);

        await completion({
            text,
            index: 0,
            expectedItems: ['A', 'B']
        });

        await completion({
            text,
            index: 1,
            expectedItems: ['A']  // would expect ['A', '.'] here
        });

        await completion({
            text,
            index: 2,
            expectedItems: ['.']  // would expect ['.', 'A', 'B'] here
        });

        await completion({
            text,
            index: 3,
            expectedItems: ['A', 'B']
        });
    });