ExpediaGroup / waggle-dance

Hive federation service. Enables disparate tables to be concurrently accessed across multiple Hive deployments.
Apache License 2.0
273 stars 76 forks source link

Blocks other threads when initialize HiveConf #325

Closed wxplovecc closed 3 months ago

wxplovecc commented 3 months ago

Describe the bug When the request QPS with 1000+, Blocks other threads when initialize HiveConf.

To Reproduce 企业微信截图_17204612901695

Expected behavior May be we should add some cache.

    HiveConfFactory confFactory = new HiveConfFactory(Collections.emptyList(), properties);
    HiveConf conf = hiveConfs.computeIfAbsent(uris, t-> confFactory.newInstance());
    return defaultMetaStoreClientFactory
        .newInstance(conf, "waggledance-" + name, DEFAULT_CLIENT_FACTORY_RECONNECTION_RETRY,
            connectionTimeout);

Logs Please add the log output from Waggle Dance when the error occurs, full stack traces are especially useful. It might also be worth looking in the Hive metastore service log files to see if anything is output there and including that too.

Versions (please complete the following information):

Additional context Add any other context about the problem here.

patduin commented 3 months ago

nice find, yeah they only need to be created on application startup, so some sort of cache or eager initialisation would be fine.

Would you consider raising a PR?

wxplovecc commented 3 months ago

fixed