search

hit-lacus / kylin

Kylin Repo
Apache License 2.0
0 stars 0 forks source link

Verify Kylin 3.1.0 on EMR #1

Open hit-lacus opened 4 years ago

hit-lacus commented 4 years ago

Test report for EMR related issue

Check list

Total estimate

Note

If you find some unexpected and unrelated error/mistake, please DO report it if it is truly a mistake, I think we may research and fix it in the future.

If you find background information/root cause analysis is not complete or ambiguous, please try to contact to author or do a quick research and record what you find. It is a good chance to learn something interesting.

If you find it is not easy to design testcase, please notify release manager.

Following issues are all for EMR related, let me verify them in one time.

hit-lacus commented 4 years ago

Understand background

These five issues aims to fix issue which related to

  1. HiveSink of System Cube support in EMR
  2. Realtime-OLAP running on EMR with S3 as working-dir.
  3. Support Glue configured Hive metadata store.
hit-lacus commented 4 years ago

Design test case

Prepare

Test

https://github.com/hit-lacus/hit-lacus.github.io/issues/81

hit-lacus commented 4 years ago

1

hit-lacus commented 4 years ago

2

hit-lacus commented 4 years ago

Prepare the EMR

# Create a EMR cluster which use glue as metadata of Hive, and S3 as storage layer of HBase.
aws emr create-cluster --applications Name=Hadoop Name=Hive Name=Pig Name=HBase Name=Spark Name=Sqoop Name=Tez  Name=ZooKeeper \
    --tags 'Cost Center=OS' 'Project=Apache-Kylin-Release-Test' 'Owner=xiaoxiang.yu@kyligence.io' \
    --ec2-attributes '{"KeyName":"XiaoxiangYu","InstanceProfile":"EMR_EC2_DefaultRole","SubnetId":"subnet-988541f1","EmrManagedSlaveSecurityGroup":"sg-9a1555f3","EmrManagedMasterSecurityGroup":"sg-9e1656f7"}' \
    --release-label emr-5.30.0 \
    --log-uri 's3n://aws-logs-472319870699-cn-northwest-1/elasticmapreduce/' \
    --instance-groups '[{"InstanceCount":2,"EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"SizeInGB":50,"VolumeType":"gp2"},"VolumesPerInstance":1}]},"InstanceGroupType":"CORE","InstanceType":"m5.xlarge","Name":"Worker Node"},{"InstanceCount":1,"EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"SizeInGB":100,"VolumeType":"gp2"},"VolumesPerInstance":1}]},"InstanceGroupType":"MASTER","InstanceType":"m5.xlarge","Name":"Master Node"}]' \
    --configurations '[{"Classification":"hbase","Properties":{"hbase.emr.storageMode":"s3"}},{"Classification":"hbase-site","Properties":{"hbase.rootdir":"s3://xiaoxiang-yu/hbase/data/20200609","hbase.rpc.timeout": "3600000"}},{"Classification":"hive-site","Properties":{"hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"}},{"Classification":"mapred-site","Properties":{"mapreduce.map.java.opts":"-Xmx2458m","mapreduce.reduce.java.opts":"-Xmx4916m","mapreduce.map.memory.mb":"3072","mapreduce.reduce.memory.mb":"6144"}},{"Classification":"yarn-site","Properties":{"yarn.nodemanager.resource.cpu-vcores":"4","yarn.app.mapreduce.am.resource.mb":"6144","yarn.scheduler.maximum-allocation-mb":"12288","yarn.nodemanager.resource.memory-mb":"12288"}}]' \
    --auto-scaling-role EMR_AutoScaling_DefaultRole \
    --ebs-root-volume-size 80 \
    --service-role EMR_DefaultRole \
    --enable-debugging \
    --name 'Kylin-Open-Source' \
    --scale-down-behavior TERMINATE_AT_TASK_COMPLETION \
    --emrfs Consistent=true \
    --region cn-northwest-1

Prepare data

image

hit-lacus commented 4 years ago

Start Kylin on EMR


kylin.metadata.url=kylin_metadata_xiaoxiang@hbase
kylin.env.hdfs-working-dir=s3://xiaoxiang-yu/kylin/kylin-3.1/
kylin.storage.hbase.cluster-fs=s3://xiaoxiang-yu
kylin.source.hive.redistribute-flat-table=false
kylin.source.hive.metadata-type=gluecatalog

kylin.engine.spark-conf.spark.eventLog.dir=s3\://xiaoxiang-yu/kylin/spark-history
kylin.engine.spark-conf.spark.history.fs.logDirectory=s3\://xiaoxiang-yu/kylin/spark-history

kylin.query.max-return-rows=10000000
kylin.engine.spark-conf.spark.yarn.archive=s3://xiaoxiang-yu/package/kylin_binary_AWS_GLUE/spark-libs.jar

kylin.storage.hbase.table-name-prefix=TEMP_
kylin.web.timezone=GMT+8
kylin.stream.event.timezone=GMT+8
kylin.metrics.monitor-enabled=true
kylin.metrics.reporter-job-enabled=true
kylin.metrics.reporter-query-enabled=true
kylin.web.dashboard-enabled=true

kylin.metric.subject-suffix=XXYU_SYSTEM

image

hit-lacus commented 4 years ago

Fix external metadata store issue

http://jtdz-solenoids.com/stackoverflow_/questions/44066559/external-hive-metastore-issue-in-emr-cluster?answertab=oldest

image

hit-lacus commented 4 years ago

Prepare hive table and kafka sample data

hit-lacus commented 4 years ago

Build Cube with COUNT_DISTINCT measure

hit-lacus commented 4 years ago

Spark Build Error

20/06/10 10:21:20 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 216.5 KB, free 1028.3 MB)
20/06/10 10:21:20 ERROR KylinKryoRegistrator: failed to load class
java.lang.ClassNotFoundException: org.apache.kylin.shaded.com.google.common.collect.EmptyImmutableList
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:264)
    at org.apache.kylin.engine.spark.KylinKryoRegistrator.addClassQuitely(KylinKryoRegistrator.java:322)
    at org.apache.kylin.engine.spark.KylinKryoRegistrator.registerClasses(KylinKryoRegistrator.java:110)
    at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$6.apply(KryoSerializer.scala:137)
    at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$6.apply(KryoSerializer.scala:137)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
    at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:137)
    at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:329)
    at org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:314)
    at org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:223)
    at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:288)
    at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:127)
    at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:88)
    at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
    at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
    at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1489)
    at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1203)
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:1111)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$submitWaitingChildStages$6.apply(DAGScheduler.scala:924)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$submitWaitingChildStages$6.apply(DAGScheduler.scala:923)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
    at org.apache.spark.scheduler.DAGScheduler.submitWaitingChildStages(DAGScheduler.scala:923)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1472)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2261)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2213)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2202)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
20/06/10 10:21:20 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 45.1 KB, free 1028.3 MB)
hit-lacus commented 4 years ago

System Cube HiveSink

2020-06-10T10:32:52,751 ERROR [metrics-blocking-reservoir-scheduler-0] org.apache.kylin.metrics.lib.impl.hive.HiveReservoirReporter - null
java.lang.UnsupportedOperationException
    at com.amazon.ws.emr.hadoop.fs.s3n2.S3NativeFileSystem2.append(S3NativeFileSystem2.java:151) ~[emrfs-hadoop-assembly-2.40.0.jar:?]
    at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:1181) ~[hadoop-common-2.8.5-amzn-6.jar:?]
    at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.append(EmrFileSystem.java:300) ~[emrfs-hadoop-assembly-2.40.0.jar:?]
    at org.apache.kylin.metrics.lib.impl.hive.HiveProducer.write(HiveProducer.java:251) ~[kylin-metrics-reporter-hive-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
    at org.apache.kylin.metrics.lib.impl.hive.HiveProducer.send(HiveProducer.java:145) ~[kylin-metrics-reporter-hive-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
    at org.apache.kylin.metrics.lib.impl.hive.HiveReservoirReporter$HiveReservoirListener.onRecordUpdate(HiveReservoirReporter.java:144) [kylin-metrics-reporter-hive-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
    at org.apache.kylin.metrics.lib.impl.BlockingReservoir.notifyListenerOfUpdatedRecord(BlockingReservoir.java:125) [kylin-core-metrics-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
    at org.apache.kylin.metrics.lib.impl.BlockingReservoir.onRecordUpdate(BlockingReservoir.java:113) [kylin-core-metrics-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
    at org.apache.kylin.metrics.lib.impl.BlockingReservoir.access$300(BlockingReservoir.java:39) [kylin-core-metrics-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
    at org.apache.kylin.metrics.lib.impl.BlockingReservoir$ReporterRunnable.run(BlockingReservoir.java:189) [kylin-core-metrics-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
[root@ip-172-31-9-20 apache-kylin-3.1.0-SNAPSHOT-bin]# cat logs/kylin.out | grep  supportAppend
2020-06-10T10:06:11,029 INFO [metrics-blocking-reservoir-scheduler-0] org.apache.kylin.metrics.lib.impl.hive.HiveProducer - For hdfs://ip-172-31-9-20.cn-northwest-1.compute.internal:8020, supportAppend was set to true
2020-06-10T10:32:52,246 INFO [metrics-blocking-reservoir-scheduler-0] org.apache.kylin.metrics.lib.impl.hive.HiveProducer - For hdfs://ip-172-31-9-20.cn-northwest-1.compute.internal:8020, supportAppend was set to true

image

hit-lacus commented 4 years ago

Shrunken dictionary

image

cp /etc/hive/conf/hive-site.xml $KYLIN_HOME/conf/
  <property>
    <name>fs.defaultFS</name>
    <value>s3://xiaoxiang-yu/kylin/kylin-3.1/</value>
  </property>
hit-lacus commented 4 years ago

base_2020_06_10_10_15_31.zip

hit-lacus commented 4 years ago

Summary