Missing columns for 'v2/search/basic' API from Atlas

[saurabh-shinde]

Hi, We are trying to enable Apache Atlas as a catalog using this connector. But we are noticing issue with query results for 'v2_search_basic' table from this catalog.

But we are unable to see all columns from response in query results

Connector version - 1.31 API specification

[nineinchnick]

Which columns are missing?

[saurabh-shinde]

We are not seeing entities in query results, which is the crucial one as it has details for entities that match search results API response:

queryType
searchParameters
queryText
entities
approximateCount

Query result:

full_text_result
type_name
approximate_count
offset
exclude_deleted_entities
query
classification_req
sort_by
query_type
type
classification
search_parameters
next_marker
marker
limit
referred_entities
attributes
sort_order
query_text

[nineinchnick]

This is because entities are an array of AtlasEntityHeader and it's defined as:

      "AtlasEntityHeader": {
        "title": "AtlasEntityHeader",
        "description": "An instance of an entity - like hive_table, hive_database.",
        "allOf": [
          { "$ref": "#/components/schemas/AtlasStruct" },
          {
            "type": "object",
            "properties": {
              "classificationNames": {
                "type": "array",
                "description": "",
                "items": { "type": "string" }
              },
              "classifications": {
                "type": "array",
                "description": "",
                "items": { "$ref": "#/components/schemas/AtlasClassification" }
              },
              "displayText": { "type": "string", "description": "" },
              "guid": { "type": "string", "description": "" },
              "isIncomplete": { "type": "boolean", "description": "" },
              "labels": {
                "type": "array",
                "description": "",
                "items": { "type": "string" }
              },
              "meaningNames": {
                "type": "array",
                "description": "",
                "items": { "type": "string" }
              },
              "meanings": {
                "type": "array",
                "description": "",
                "items": {
                  "$ref": "#/components/schemas/AtlasTermAssignmentHeader"
                }
              },
              "status": { "$ref": "#/components/schemas/Status" }
            }
          }
        ]
      },

There's a allOf union type that's not supported. There's an existing issue to support fields like this: #52

[nineinchnick]

I've got a workaround in #66 to treat such union types as string