why not just submit a patch to postgresql project make raw_parser exported to libpq

[ChaosEternal]

Apologies that it may not be a proper issue for this project. But I just want to know whether anyone has tried to push the changes to the postgresql project.

[lfittl]

@ChaosEternal Its a good question - I think this is out of scope for libpq to be honest, and the likely response would be "who needs this, really?".

The other issue is that this library does a lot of hacks to trick the PostgreSQL server code into acting like a library, including stubbing methods that are only available when running as a backend. It would be quite invasive to patch up things properly, I think.

That said, if you have resources available to work on this, I'd be happy to help.

[davidfetter]

I suspect that a good many people would really like to have this shipped with PostgreSQL. If it were, would you see requiring a live DB connection as a reasonable part of it?

[lfittl]

@davidfetter That is a good question.

I think there is value in offering this in core on a running server (i.e. requiring a live DB connection), and it would likely be much easier to get that reviewed and committed.

That said, my personal use case has usually been to use this to process data from another Postgres server (in particular as part of the query processing pipeline of pganalyze), and its been convenient to not require a Postgres connection for this purpose, but instead just being able to call the library within the worker process.

My ideal solution would be something like libpqparser that is independent of the server in terms of installation and usage, although the code would reside in core.

Note that another part that has been used quite a lot is deparsing (the Node.js library has the most complete version of it, the Ruby version has quite okay version of it too). When moving this to core, one question is whether it'd be possible to re-use existing infrastructure to handle the deparsing, instead of having to re-invent the wheel in the library.

What do you think makes sense?

[davidfetter]

What would be ideal for me may not be practical as a goal. This would be:

• A parser which has current access to the catalog of the database in question, i.e. has a live connection to same. This would have some nice properties, among which would be to:
  • See database objects that actually exist apart from what's in an empty database. The attempt to use objects not present could be set to be an error.
  • See only things the role was permitted to see. Again, the attempt to use objects not visible could be set to be an error.

This could degrade to:

• A stand-alone parser which would know about PostgreSQL's grammar and supplied objects

While I'm wishing for unicorns, it would be nice if there were some way to extend it to include:

• contrib extensions
• PLs

All this makes it a fit candidate for inclusion in core, as it wouldn't be a lot of fun to play catch-up each major version.

[lfittl]

@davidfetter Nice, I think that makes sense - ideally we could still have the raw parser without analysis accessible as a standalone library, but this could complement each other well.

I'm trying to allocate some time for more Postgres core work in the v12 cycle, so I might try to pick it up then (but can't promise anything). Of course happy to review any patches in case you end up finding time to work on this.

[mitar]

Based on https://wiki.postgresql.org/wiki/Query_Parsing it looks like there is some interest to getting this into PostgreSQL.

[sgreene570]

cc @jmo-qap