search

trinodb / trino

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
https://trino.io
Apache License 2.0
10.36k stars 2.98k forks source link

Improve startup time of HDP3 test environments #4091

Closed findepi closed 3 years ago

findepi commented 4 years ago 
presto-product-tests-launcher/bin/run-launcher env up --environment singlenode-hdp3
...
INFO    testcontainers-lifecycle-0  🐳 [prestodev/centos7-oj11:28]   Container prestodev/centos7-oj11:28 started in PT37.316069S
INFO    testcontainers-lifecycle-1  🐳 [prestodev/hdp3.1-hive:28]    Container prestodev/hdp3.1-hive:28 started in PT1M44.238856S
wendigo commented 4 years ago

I'd fix in HDP: hadoop-hdfs-namenode.log:2020-06-19 14:57:11,745 - INFO [main:BlockManagerSafeMode@602] - STATE* Safe mode ON. which gives 30s penalty on startup

findepi commented 4 years ago

My guess is this has something to do with namenode starting in safe mode, which delays HiveServer2 start and opening port 10000. We could try to force safe mode exit.

wendigo commented 4 years ago

Yeah :)

wendigo commented 4 years ago

We could try setting:

    <property>
        <name>dfs.safemode.threshold.pct</name>
        <value>0</value>
    </property>

in hdfs-site.xml

wendigo commented 4 years ago 
2020-06-19T11:31:37.837+0200    INFO    testcontainers-lifecycle-1  🐳 [prestodev/hdp3.1-hive:28]    Container prestodev/hdp3.1-hive:28 started in PT25.876362S
wendigo commented 4 years ago

https://github.com/prestosql/presto/pull/4094

wendigo commented 4 years ago

https://github.com/prestosql/docker-images/pull/60