quintoandar / hive-metastore-client

A client for connecting and running DDLs on hive metastore.
Apache License 2.0
52 stars 15 forks source link

[DAE-145] Change the default values for create table event #62

Closed kenjihiraoka closed 3 years ago

kenjihiraoka commented 3 years ago

Why? :open_book:

We've been using the hive-metastore listener to track all DDLs running on the server that generates a payload sent to the Kafka topic in order for the Apache Atlas can read them and create the objects in your ecosystem.

Using the lib we faced a problem with the creation of tables based on Json schema, actually, the creation of the table worked as expected, but the listener could not get the json schema, given the following error:

2021-08-06T18:26:43,438 ERROR [pool-8-thread-2] metadata.Table: Unable to get field from serde: org.apache.hive.hcatalog.data.JsonSerDe
java.lang.NullPointerException: null

In the first shot, that error looks like something related to the listener code, but after realizing plenty of tests with many kinds of changes in the listener, I could not find any idea of the reason for this specific error: a mere NullPointer :disappointed:

After that, I tried to get the object created by the listener and did a comparison with the object saved in the hive-metastore database through the hive-metastore client, and I noticed that there were different objects for the same table, so I could see that some default values for TableBuilder and SerdeInfoBuilder were the reason for the specific error above.

What? :wrench:

Type of change :file_cabinet:

How everything was tested? :straight_ruler:

Ran the create table event with those hard-coded attributes and checked the hive-metastore log that got the json schema.

Checklist :memo:

sonarcloud[bot] commented 3 years ago

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication