gRest: Convert views to RPC, simplify whitelisting URL process

[rdlrt]

Describe the Feature Request

As part of Koios gRest Node setup, we had a whitelisting requirement to ensure that cache tables were not directly query-able from HAProxy (primarily due to accuracy and lock concerns).

By default, PostgREST serves RPC endpoints with /rpc prefix, while views/tables are served at root (/) in a default configuration. Currently , the conversion from /api/v0 to the right endpoint on PostgREST was achieved by creating a grestrpcs file as part of setup-grest.sh , which would:

• scan koiosapi.yaml downloaded from the API endpoint
• check out for comment #RPC and create a URL whitelist file grestrpcs, which was used as ACL in haproxy config to do a replace for /api/v0/rpc with /rpc if endpoint was RPC , otherwise replace /api/v0 with / . This ensured that paths are resolved correctly going from haproxy to PostgREST.

This was early implementation but as we learn, I think we can improve this flow a bit. To clarify, downsides of this process:

• When a new endpoint is added across a version, this requires a restart of haproxy server - which can be easily missed by instance providers (especially when not clarified in release notes or upgrade instructions 🙁 ). This could be especially problematic when going through development testing as new endpoints will not be available until an entry is added to grestrpcs and haproxy restarted.
• adds a new file dependency in haproxy config as well as grest-poll.sh/setup-grest.sh to maintain, that we should be OK to do without
• It still allows querying tables/views that are not defined as RPC from instance provider to monitoring servers (assuming that's the only port enabled).

The proposal is to simplify this process:

• Convert all views into new RPC endpoints (ensure the only need to update specs is removal of #RPC keyword). Thus - avoids having a change for consumers.
• Make control_table queryable using a RPC as well.
• Update haproxy configs in setup-grest.sh to remove references of grestrpcs. Simplify regex to expose all RPCs by default, and no table/views - this will ensure that none of the cache tables, experimentation, etc is exposed, only the RPCs created in grest schema are exposed, which also makes it easier to dynamically add entries without having to wait for changes in external file (worse case - will require schema reloading in PostgREST which should be seamless)
• Update setup-grest.sh and grest-poll.sh to remove references for creation/query of grestrpcs file(s)

I am happy to have this in koios-1.0.9 if we want to give it some thought and time to discuss , to keep koios-1.0.8 release simple.

[dostrelith678]

I agree for this to be included in version 1.0.9.

[rdlrt]

Played with this idea a bit as part of asset_list endpoint, the trouble is for function wherein the table gets processed prior to applying parameters. Ideally we'd want to in that case be able to use limit/offset parameters passed to the URL (which will require us to maintain a single commit addition to PostgREST).

Removing vNext milestone for now, as it's not gonna be as seamless as we originally thought.

As regards haproxy config, if it comes up - the obvious solution for those familiar with config might be 'why not add ACL into the config?'

This was considered, but this would mean every time a view is to be added, we would require asking instance providers to update their haproxy config (though it is still simply running a shell script, it gets complicated to retain user customisations made on base config). Having this dumped to seperate file that is auto-generated during setup-grest ensures that file can be overwritten without any impact.

Once above dilemma is tackled and agreed upon, the hope is to get rid of views completely

[rdlrt]

Having played around with this one, it looks like whilst this is possible for simple functions, there are quite a few assumptions (see here , each of these will need to have a handle for special case, and needs to be integrated on all instances), and makes deployment of setup-grest.sh (deployment) and grest-poll.sh (verification of existence of the condition) quite complicated, which is actually against the whole intent of wrtiting this issue in the first place.

On the contrary, with cardano-community/guild-operators#1572 , the haproxy will anyways get simplified (just that we wont be able to get rid of grestrpcs, but for instance provider - they'd no longer need to worry about monitoring layer additions in a default config, that role will be onboarded by monitoring instances).

Hence, closing this one