search

odpi / egeria-connector-sas-viya

Egeria connector for SAS Information Catalog
Apache License 2.0
17 stars 9 forks source link
egeria

readme

SAS-Egeria Connector

Overview

The purpose of this project is to create an Egeria connector for SAS Information Catalog.

Much of this code is based off of the Atlas-Egeria connector, which you can find here.

Starting Egeria Server with Connector

  1. Run gradlew clean build install in this repository
  2. Bring up the docker containers in src/test/resources/compose with the command docker-compose -f egeria-tutorial.yaml up.
    If you'd prefer to only bring up the connector by itself, use docker-compose -f connector-minimal.yaml up instead. Make sure you have allocated sufficient RAM to the Docker image. The default of 2 GB will almost certain result in out of memory conditions (e.g. error 137 from compose_ui_1). Try 6 GB.
  3. Bring up the Jupyter notebook at http://localhost:18888 and run through all of the cells in egeria-server-config.ipynb after changing the Catalog connection settings for the connector

    Alternatively, you can use the Postman collection in src/test/resources to configure and start the server if you aren't interested in starting the other servers and setting up cohorts. (See "Testing via Postman")

Testing via UI

After starting up the server(s), you can access the Egeria UI at https://localhost:18443 with the username garygeeke and password admin.

Then, in Type or Repository explorer use the server name test , and the server URL root http://connector:8080 to access the Connector via the UI.

Testing via Postman

To have more control over your requests and to be able to see the unaltered response, use Postman to make your Egeria requests instead. Import the Egeria collection mentioned in step 3 above.

After importing this collection into Postman, you need to set three global or collection variables:

Then, to configure the server, open the first request: "00.0 - Set Catalog Connector". You will need to modify the connection properties to point at the Catalog server you will be connecting to. Then send the request.

Once the server is configured, start it by sending the second request: "00.1 - Start Server".

Now that the server is up and running, you are free to make any of the other requests. Note that many of these are not yet implemented.

Also keep in mind that you will need to change the GUIDs in the requests to Catalog instance IDs for the Catalog server you have access to.

Type mappings

Mappings can be found in src/main/resources/TypeDefMappings.json.

This file contains a list of mapping objects that tell the connector how to map SAS Catalog objects into Egeria objects.

The format for a mapping object is as follows.

For entities:

{
  "sasCat": "<Catalog definition name>",
  "prefix": "<ID Prefix if one catalog type maps to multiple types>",
  "omrs": "<Egeria type name>",
  "propertyMappings": [
    {
      "sasCat": "[instance|definition|attribute|constant].<Catalog property name>",
      "omrs": "<Egeria property name>"
    }, 
    {
      "sasCat": "instance.name",
      "omrs": "qualifiedName"
    },
    {
      "sasCat": "instance.label",
      "omrs": "displayName"
    }
  ]
}

If the Catalog entity is a reference: use "reference." as the definition name.

For relationships:

{
  "sasCat": "<Catalog rel. definition name>",
  "omrs": "<Egeria rel. type name>",
  "endpointMappings": [
    {
      "sasCat": "[OPTIONAL] <Catalog entity def. name (must be mapped)>",
      "prefix": "[OPTIONAL] <Prefix to use for this entity (must match entity mapping prefixes)>",
      "omrs": "<Egeria name for this endpoint>"
    },
    {
      "sasCat": "[OPTIONAL] <Catalog entity def. name (must be mapped)>",
      "prefix": "[OPTIONAL] <Prefix to use for this entity (must match entity mapping prefixes)>",
      "omrs": "<Egeria name for this endpoint>"
    }
  ]
}

The only required field for endpoint mappings is "omrs".

You can also specify "prefix" without specifying "sasCat". This will use the mapping for that entity type with the specified prefix, but will allow different entity types to be used.

For example, if one of the endpoints is of type "SchemaType", and you had entity mappings with prefix "RTT" for both RelationalTableType and RelationalColumnType (which both extend SchemaType), you could specify the prefix "RTT" in the relationship mapping and be able to map a Catalog table and Catalog column, assuming these map to RelationalTableType and RelationalColumnType respectively, to that endpoint.

Catalog definition names and instance attributes can be found in archive/definition_archive.go in the Catalog repository.

For the Catalog property name, you can use one of four prefixes:

To send requests to Catalog, you can use the Catalog Postman collection in src/test/resources. You will need to set the collection variable ip to the IP of the Catalog server you are using.

Egeria type and property names can most easily be found by using Types Explorer in the Egeria UI.
To do this, go to the UI and login as before. Go to Types Explorer, and use cocoMDS1 as the server name and http://datalake:8080 as the server URL root.
From there, you will see a list of the Egeria types and the attributes on them.

Debugging

After starting the server with the steps above, you can attach a debugger remotely via localhost:17084.

IntelliJ

Contributing

We welcome your contributions! Please read CONTRIBUTING.md for details on how to submit contributions to this project.

License

This project is licensed under the Apache 2.0 License.