search

TOSIT-IO / tdp-collection

Ansible collection to deploy the components of TDP
Apache License 2.0
21 stars 19 forks source link

Upgrade YARN Application Timeline Service to v2 #623

Open leopaul36 opened 1 year ago

leopaul36 commented 1 year ago

We could upgrade YARN Application Timeline Service to v2. This versions runs in YARN and allows HA / distributed storage.

Docs: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html

See https://github.com/TOSIT-IO/tdp-collection/pull/612

dapicard commented 1 year ago

Hi, I'm working on this issue and I face a dependency version problem. Timeline-server V2 uses Hbase as its persitence backend so we have to create some tables using a tool named org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator.

This tool is built using Hbase 1.2.6 but we use HBase 2.1.10. So when I try to create the tables, I encounter a linking issue :

HADOOP_OPTS=-Djava.security.auth.login.config=/etc/hbase/conf/krb5JAASClient.conf HADOOP_CLASSPATH=/opt/tdp/hbase/lib/*:/opt/tdp/hadoop-3.1.1-TDP-0.1.0-SNAPSHOT/share/hadoop/yarn/timelineservice/* hadoop org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -create -skipExistingTablell
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Admin.createTable(Lorg/apache/hadoop/hbase/HTableDescriptor;[[B)V
        at org.apache.hadoop.yarn.server.timelineservice.storage.entity.EntityTableRW.createTable(EntityTableRW.java:121)
        at org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator.createAllTables(TimelineSchemaCreator.java:307)
        at org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator.createAllSchemas(TimelineSchemaCreator.java:277)
        at org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator.main(TimelineSchemaCreator.java:146)
...

There is some solutions I can test :

What should I do ?

dapicard commented 1 year ago

I give a try to update hbase-client to 2.1.10 in the TOSIT hadoop package (see https://github.com/TOSIT-IO/hadoop/issues/3) and the schema creation works fine. So I will finish to integrate Timeline Server v2 taking into consideration that hadoop will be built with HBase 2.1.10.