search

DataJunction / dj

A metrics platform.
http://datajunction.io
MIT License
34 stars 15 forks source link

Fix the AST parser's support for hints and comments #1026

Open shangyian opened 4 months ago

shangyian commented 4 months ago

At the moment, the AST does not support parsing of Spark hints (nor comments in general). We're likely missing some functions from the antlr parser that handles hints, since the hints are there in the grammar.

This unit test illustrates the issue:

@pytest.mark.asyncio
async def test_ast_hints(session: AsyncSession):
    """
    Test parsing a query with hints
    """
    query_str = """SELECT /*+ REPARTITION(c) */ c, b, a FROM t"""
    query = parse(str(query_str))
    assert str(parse(str(query))) == str(parse(str(query_str)))
    exc = DJException()
    ctx = ast.CompileContext(session=session, exception=exc)
    await query.compile(ctx)
    assert not exc.errors