Table Type seems to be ignore on ttypes.Table

[bandianconde]

Hello,

I tried to use the object ttypes.Table as seen in tests/integration/test_metastore.py When I create a table using ttypes.Table and set tableType to be 'EXTERNAL_TABLE' , and then use get_table the tableType is returned as MANAGED_TABLE. ` with HMS.create(host='localhost', port=9083) as hm: table = ttypes.Tables( tableName='table_name', dbName='db_name', tableType='EXTERNAL_TABLE', ..... ) hm.client.create(table) new_table = hm.client.get_table('db_name', 'table_name') assert new_table.tableType == 'MANAGED_TABLE'

`

[bandianconde]

Hello @criccomini @agrueneberg , Do yo need more details ?

[criccomini]

Lemme take a look right now. :)

[criccomini]

@bandianconde, one point of clarification: are you setting tableType to EXTERNAL_TYPE or EXTERNAL_TABLE?

[criccomini]

Looks like this is a known issue:

https://github.com/recap-build/pymetastore/blob/main/tests/integration/test_metastore.py#L429

[criccomini]

So, some digging. The call stack is as follows:

test_metastore.py line 414:
    table = hms.get_table("test_db", "test_table")

metastore.py line 390:
        table: Table = self.client.get_table(database_name, table_name)

# At this point, we're in code-gen land and have no control over what's happening.
ThriftHiveMetastore.py line: 3271:
        self.send_get_table(dbname, tbl_name)

Later in get_table, starting at line 591, we have:

        if table.tableType is not None:
            if isinstance(table.tableType, str):
                table_type = table.tableType
            else:
                raise TypeError(
                    f"Expected tableType to be str, got {type(table.tableType)}"
                )
        else:
            raise TypeError(
                f"Expected tableType to be str, got {type(table.tableType)}"
            )

I have to believe that tableType is coming back as either None or EXTERNAL_TABLE. This seems like an issue with the metastore AFAICT.

Let me do a little more digging.

[criccomini]

@bandianconde, also, what metastore are you using? Hive? Glue?

And how was the table created? Via pymetastore? Or via some other means?

[criccomini]

This looks relevant: https://github.com/apache/hive/pull/1537

Unfortunately, it's not fixed and closed as stale.

[bandianconde]

@bandianconde, one point of clarification: are you setting tableType to EXTERNAL_TYPE or EXTERNAL_TABLE?

EXTERNAL_TABLE it was a typo sorry

[criccomini]

@bandianconde, also, what metastore are you using? Hive? Glue?

And how was the table created? Via pymetastore? Or via some other means?

@bandianconde ^

[bandianconde]

@bandianconde, also, what metastore are you using? Hive? Glue?

And how was the table created? Via pymetastore? Or via some other means?

I use hive. I bypass pymetastore and use directly ttypes.Table as show in the tests

[criccomini]

What version of HMS?

[bandianconde]

Version 3.0.0

[bandianconde]

@criccomini I have seen here https://github.com/quintoandar/hive-metastore-client/blob/main/hive_metastore_client/hive_metastore_client.py#L204 that they had the same problem and fix it with by adding table table.parameters = {"EXTERNAL": "TRUE"} before creating the table. I tested it and it's working fine

[criccomini]

Nice. That's a good find.

I'm first upgrading the tests to use Hive 3.1.3. I want to see if HMS fixes the issue in a future version. HMS 3.0 is quite old.

[criccomini]

OK, just verified the issue exists even for HMS 3.1.3 Let's move forward with the table.paramaters patch you suggest.

[criccomini]

Working on fix: https://github.com/recap-build/pymetastore/pull/38

[bandianconde]

Thanks, I have seen your PR, it should be fine. However it looks like on some tests you need to add this new parameter as expected

[criccomini]

Yep workin on it =)

[criccomini]

Merged! No new release is needed since this is just a change to the way users should invoke create_table. LMK if you want a release anyway.

[bandianconde]

On my side I don't use a new release as use directly the ttypes.Table. Thanks !!