feat: split_with_parser, split_with_scanner

[SKalt]

I'm noticing some weirdness in the split_with_scanner example. Malformed tokens seem to be getting dropped:

assert_eq!(
    pg_query::split_with_scanner("select 1; asdf; select 3;").unwrap(),
    vec!["select 1", "asdf", "select 3"],
); // fails: actually produces ["select 1", "select 3"]

Is this expected?

Resolves #5.

[lfittl]

I'm noticing some weirdness in the split_with_scanner example. Malformed tokens seem to be getting dropped:

assert_eq!(
    pg_query::split_with_scanner("select 1; asdf; select 3;").unwrap(),
    vec!["select 1", "asdf", "select 3"],
); // fails: actually produces ["select 1", "select 3"]

Is this expected?

Yup, that's expected, if the statement only contains invalid tokens, we skip it here:

https://github.com/pganalyze/libpg_query/blob/13-latest/src/pg_query_split.c#L113

We could change that behavior, its not something that anyone relies on today, to my knowledge. Would it be helpful for your use case?

[SKalt]

Cool, I'll update my comment on the function. I don't depend on that function, but I did have a use-case for split_with_scanner: I wanted to split the postgres regression tests into individual statements to build a SQL test corpus (skalt/sql_parser_tests). The problem was using split_with_scanner ended up mixing in psql meta-commands, so I bailed and wrote a nom parser that did what I wanted (skalt/pqsl_splitter).

[lfittl]

Cool, I'll update my comment on the function. I don't depend on that function, but I did have a use-case for split_with_scanner: I wanted to split the postgres regression tests into individual statements to build a SQL test corpus (skalt/sql_parser_tests). The problem was using split_with_scanner ended up mixing in psql meta-commands, so I bailed and wrote a nom parser that did what I wanted (skalt/pqsl_splitter).

Great, and neat idea!

I'll leave the Rust code review to @seanlinsley, since he's been maintaining most of the library, but looks good from a quick glance :)

[SKalt]

First-time contributors need a maintainer to approve running workflows

@seanlinsley would you be willing to approve workflows on this PR so we're all looking at the same unit test output?

[seanlinsley]

Thanks for submitting this PR @SKalt!