trinodb / trino

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
https://trino.io
Apache License 2.0
10.27k stars 2.95k forks source link

when trino connecting to hive metastore,It's always connect in the same order #10305

Closed lurnagao-dahua closed 2 months ago

lurnagao-dahua commented 2 years ago

Hi, i have a quesion.

when trino connecting to hive metastore,It's always connect in the same order

public ThriftMetastoreClient createMetastoreClient(Optional<String> delegationToken)
        throws TException
{
    Comparator<Backoff> comparator = Comparator.comparingLong(Backoff::getBackoffDuration)
            .thenComparingLong(Backoff::getLastFailureTimestamp);
    List<Backoff> backoffsSorted = backoffs.stream()
            .sorted(comparator)
            .collect(toImmutableList());

    TException lastException = null;
    for (Backoff backoff : backoffsSorted) {
        try {
            return getClient(backoff.getAddress(), backoff, delegationToken);
        }
        catch (TException e) {
            lastException = e;
        }
    }

    List<HostAndPort> addresses = backoffsSorted.stream().map(Backoff::getAddress).collect(toImmutableList());
    throw new TException("Failed connecting to Hive metastore: " + addresses, lastException);
}

and the getClient is:

private ThriftMetastoreClient getClient(HostAndPort address, Backoff backoff, Optional<String> delegationToken)
        throws TException
{
    ThriftMetastoreClient client = new FailureAwareThriftMetastoreClient(clientFactory.create(address, delegationToken), new Callback()
    {
        @Override
        public void success()
        {
            backoff.success();
        }

        @Override
        public void failed(TException e)
        {
            backoff.fail();
        }
    });
    if (!isNullOrEmpty(metastoreUsername)) {
        client.setUGI(metastoreUsername);
    }
    return client;
}

and the backoff.fail() is :

synchronized void fail()
{
    lastFailureTimestamp = OptionalLong.of(ticker.read());
    backoffDuration = min(backoffDuration * 2, MAX_BACKOFF);
}

when create metastore client, it will connect to metastore service sort by lastFailureTimestamp from small to big.

if one metastore service is down, then clientFactory.create function will throw TException and not going to Callbackfunction. so It's always connect in the same order.

Appreciate for help!

findepi commented 2 years ago

cc @losipiuk

nwdsslzwf commented 2 years ago

I have the same question, and I hope someone can answer it.Appreciate for help!