ExpediaGroup / waggle-dance

Hive federation service. Enables disparate tables to be concurrently accessed across multiple Hive deployments.
Apache License 2.0
266 stars 74 forks source link

Blocks other threads when initialize HiveConf #325

Closed wxplovecc closed 1 month ago

wxplovecc commented 1 month ago

Describe the bug When the request QPS with 1000+, Blocks other threads when initialize HiveConf.

To Reproduce 企业微信截图_17204612901695

Expected behavior May be we should add some cache.

    HiveConfFactory confFactory = new HiveConfFactory(Collections.emptyList(), properties);
    HiveConf conf = hiveConfs.computeIfAbsent(uris, t-> confFactory.newInstance());
    return defaultMetaStoreClientFactory
        .newInstance(conf, "waggledance-" + name, DEFAULT_CLIENT_FACTORY_RECONNECTION_RETRY,
            connectionTimeout);

Logs Please add the log output from Waggle Dance when the error occurs, full stack traces are especially useful. It might also be worth looking in the Hive metastore service log files to see if anything is output there and including that too.

Versions (please complete the following information):

Additional context Add any other context about the problem here.

patduin commented 1 month ago

nice find, yeah they only need to be created on application startup, so some sort of cache or eager initialisation would be fine.

Would you consider raising a PR?

wxplovecc commented 1 month ago

fixed