search

trinodb / trino

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
https://trino.io
Apache License 2.0
10.19k stars 2.94k forks source link

Allow connector to provision multiple catalogs #18339

Open kokosing opened 1 year ago

kokosing commented 1 year ago

There are certain connectors (like JDBC connectors) that maps remote catalog to Trino catalog. In case remote data source have multiple catalogs then user needs to setup multiple catalogs in Trino too manually. Each of them would be considered from Trino engine point of view as separate and independent catalog like they were from different connectors. With this way we are losing an opportunity of optimization like join pushdown over multiple Trino catalogs but the same connector.

Having connector to provision multiple catalogs, Trino engine would be able notice such case and potentially it would possible to allow join pushdown in such case to shared connector.

CC: @wendigo @martint @findepi @hashhar

hashhar commented 1 year ago

The other benefit would be that connectors can create catalogs for each schema in the remote database. This avoids users having to provision a new catalog and restart Trino if a new schema is added in remote system without having to implement 4 part identifiers. (e.g. Redshift, BigQuery etc.)

Praveen2112 commented 1 year ago

Almost a similar issue - https://github.com/trinodb/trino/issues/1135

huw0 commented 3 months ago

This would be really useful when paired with #21791