quintoandar / hive-metastore-client

A client for connecting and running DDLs on hive metastore.
Apache License 2.0
52 stars 15 forks source link

Program hangs after first method call #69

Closed StewartThomson closed 2 years ago

StewartThomson commented 2 years ago

Describe the bug

When calling methods in a loop, only the first call succeeds. The next subsequent call seems to hang indefinitely. Event if you use the same table in succession. This happens with any method call, not just get_partition_keys_objects

To Reproduce

Steps to reproduce the behavior:

from hive_metastore_client import HiveMetastoreClient

tables = [
  "my_table",
  "my_table",
]
with HiveMetastoreClient(
    "my_url"
) as hive_client:
  for table in tables:
    print(table)
    print(hive_client.get_partition_keys_objects("default", table))

Expected behavior

Each call should succeed in a timely manner

Environment

StewartThomson commented 2 years ago

I believe the way to fix this is to use the client as such:

from hive_metastore_client import HiveMetastoreClient

tables = [
  "my_table",
  "my_table",
]

for table in tables:
  print(table)
  with HiveMetastoreClient(
      "my_url"
  ) as hive_client:
    print(hive_client.get_partition_keys_objects("default", table))
felipemiquelim commented 2 years ago

@StewartThomson Thanks for the feedback!

Actually we've run the code for multiple calls and for diff methods as you suggested. But we did not find any issues. Attached is an example of a very simple run that returned results (in this case both cases have similar partitions, so the results are the same).

hive_metastore_client_usage

I'd double check the Hive configurations, maybe it is configured to only return one request per connection?

The way you solved the problem may work, however it is unfortunately spawning one connection per request, what can raise some issues down the road.