[feature] Trino support to Openmetadata connector

[eformat]

i was testing a Trino/Hive, Openmetadata and fabryk scenario, basically exposing an S3 file in hive via Trino, registering in Openmetadata.

But failed to get fabryk openmetadata work as it seems to currently support these mappings:

openmetadata-connector-b765554fc-vzjqv openmetadata-connector {"level":"info","connector":"OpenMetadata Connector","time":"2022-12-20T05:15:03Z","caller":"/home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:422","message":"Asset found: trino.hive.default.wine_quality"}
openmetadata-connector-b765554fc-vzjqv openmetadata-connector {"level":"trace","connector":"OpenMetadata Connector","time":"2022-12-20T05:15:03Z","caller":"/home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:433","message":"Latest version of the asset: 0.800000"}
openmetadata-connector-b765554fc-vzjqv openmetadata-connector {"level":"info","connector":"OpenMetadata Connector","time":"2022-12-20T05:15:03Z","caller":"/home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:442","message":"Succeeded in retrieving latest version of the asset"}
openmetadata-connector-b765554fc-vzjqv openmetadata-connector {"level":"error","connector":"OpenMetadata Connector","time":"2022-12-20T05:15:03Z","caller":"/home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_helper.go:686","message":"unrecognized servicetype Trino"}
openmetadata-connector-b765554fc-vzjqv openmetadata-connector {"level":"error","connector":"OpenMetadata Connector","time":"2022-12-20T05:15:03Z","caller":"/home/runner/work/openmetadata-connector/openmetadata-connector/pkg/openmetadata-connector-core/connector_service.go:207","message":"Construction of Asset Response failed"}
openmetadata-connector-b765554fc-vzjqv openmetadata-connector 2022/12/20 05:15:03 POST /getAssetInfo GetAssetInfo 320.404509ms

[cdoron]

Hi @eformat : Sorry for the late reply. I have been on vacation. You are correct: the openmetadata-connector currently does not support Trino. I will look into this in the next few days and get back to you with a more detailed response.

[cdoron]

Hi @eformat : I am experimenting with code that adds support for Trino. The code is very preliminary. You can find it here: https://github.com/fybrik/openmetadata-connector/pull/48

If you would like to try it out on fybrik, you can run:

kubectl edit deployment -n fybrik-system openmetadata-connector

and replace image: ghcr.io/fybrik/openmetadata-connector:0.3.0 with image: ghcr.io/fybrik/openmetadata-connector:0.0.0

If you like, we can set up a meeting. We would love to hear about your experience with fybrik.

Thanks, Doron, cdoron@il.ibm.com

[cdoron]

Here is an example of getAssetInfo REST request for a trino dataset:

$ curl -X POST localhost:8081/getAssetInfo -d '{"assetID": "trino.minio.tiny.customer", "operationType": "read"}' | jq
{
  "credentials": "/v1/omd-secrets/trino",
  "details": {
    "connection": {
      "Trino": {
        "catalog": "minio",
        "connectionArguments": {},
        "connectionOptions": {},
        "hostPort": "10.126.110.5:8080",
        "params": {},
        "proxies": {},
        "scheme": "trino",
        "supportsDatabase": true,
        "supportsMetadataExtraction": true,
        "supportsProfiler": true,
        "supportsQueryComment": true,
        "type": "Trino",
        "username": "admin"
      },
      "name": "Trino"
    },
    "dataFormat": "csv"
  },
  "resourceMetadata": {
    "columns": [
      {
        "name": "custkey"
      },
      {
        "name": "name"
      },
      {
        "name": "address"
      },
      {
        "name": "nationkey"
      },
      {
        "name": "phone"
      },
      {
        "name": "acctbal"
      },
      {
        "name": "mktsegment"
      },
      {
        "name": "comment"
      }
    ]
  }
}