[FEATURE] Support the MAP datatype

[joshuanapoli]

I have DynamoDB records with key-value pairs mapping id to documents. For example:

{
  "items": {
    "<uuid1>": { id: "<uuid1>", name: "Alice" },
    "<uuid2>": { id: "<uuid2>", name: "Bob" },
    "<uuid3>": { id: "<uuid3>", name: "Bob" }
  }
}

I think that the Athena MAP datatype is meant for this purpose. https://docs.aws.amazon.com/athena/latest/ug/data-types.html

I would like to define a schema using the MAP datatype, so that I can query my table using the DynamoDB

CREATE EXTERNAL TABLE mytable(
  items MAP<STRING, STRUCT<id:string, name:string>>
  )

The connectors appear to support ARRAY and STRUCT data-types, but not the Athena MAP data-type.

An alternative would be mapping the id-keyed document to an Athena STRING column. If the connector JSON-encoded the document, then I could decoded it in Athena/Presto. Unfortunately, this does not work. The DynamoDB connector formats the document content to the string column, but it is not JSON.

[burhan94]

We have this planned as a part of our Arrow upgrade. Stay tuned for a future release that will contain this feature.

[rnatarajan]

Hi @burhan94 , Since Master already has arrow upgraded, Curious about release of next version.

[C2BB]

Curious as well!

[jaysethia]

Hello @burhan94, looking forward to this feature support as well! Any updates on update to support MAP data from DynamoDB connector natively in Athena?

[rnatarajan]

Created a merge request with Example Record Handler with Map field - https://github.com/awslabs/aws-athena-query-federation/pull/734

If this code is added to a federated query connector,

• Athena throws following error when loading table metadata from Glue

Failed to get metadata for table () from lambda function due to java.lang.IllegalArgumentException: Unsupported Arrow Type [Map(false)] in Lambda Data Source

• Athena throws following error when a query is ran to select the field GENERIC_INTERNAL_ERROR: Exception while processing column LambdaColumnHandle{name='', type=map(array(varchar), array(varchar)), comment='null'}

[rnatarajan]

Created a Merge request with Example Record Hander adding second entry to Map - https://github.com/awslabs/aws-athena-query-federation/pull/739 Athena works fine with one entry in the map but the second entry is not returned.

Athena throws following error when loading table metadata from Glue Failed to get metadata for table (<table_name>) from lambda function due to java.lang.IllegalArgumentException: Unsupported Arrow Type [Map(false)] in Lambda Data Source

[slomkarafa]

Hello, +1 from my side, I tried to finish implementation of DeltaLake connector #509, but map support is not working properly as suggested.

[slomkarafa]

Do anyone knows/have any plans regarding this issue? Maybe @burhan94 ?

[sanatdeshpande1]

I have tried to implement this by not using Glue and directly going with Apache Arrow to construct schema

[slomkarafa]

hello @sanatdeshpande1, have you finished with any solution?

[henrymai]

We can enable map support after:

• Federation upgrades to Arrow 13 with this patch: https://github.com/apache/arrow/commit/cfcff49d89df220b6ec01bc33a04bbd3421f2f0d

• Addresses the TODO here (which was dependent on my Arrow change above): https://github.com/awslabs/aws-athena-query-federation/blob/ee25687039acd70012c717df1467559acdfcbcf9/athena-federation-sdk/src/test/java/com/amazonaws/athena/connector/lambda/data/helpers/FieldsGenerator.java#L117

Once those things are done we can be confident that Map support works and then we can enable it.

[matthost]

Seems like Arrow 13 is being used. Time to fix this? IMO this is a big feature for making these Connectors useful.

[hervenivon]

yes please 🙏