hit-lacus
commented
3 years ago Cluster Resource
| Key | Value |
|---|---|
| Node Memory | 64GB |
| Node Core | 16 |
| Node (Instance Type) | m5.4xlarge |
| Node Disk | 400GB * 2, SSD |
| Node Num | 4 Worker + 1 Master |
| EMR Version | emr-5.31.0 |
| Kylin Version | 4.0.0 |
| Yarn Memory | 204.80 GB |
| Yarn Core | 52 |
- Scripts
aws emr create-cluster --applications Name=Hadoop Name=Hive Name=Spark Name=ZooKeeper Name=Tez Name=Ganglia \ --tags 'Project=Kylin_Benchmark' 'Owner=xiaoxiang.yu' \ --ec2-attributes '{"KeyName":"?","AdditionalSlaveSecurityGroups":["?"],"InstanceProfile":"?","SubnetId":"?","EmrManagedSlaveSecurityGroup":"?","EmrManagedMasterSecurityGroup":"?","AdditionalMasterSecurityGroups":["?"]}' \ --release-label emr-5.31.0 \ --instance-groups '[{"InstanceCount":4,"EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"SizeInGB":400,"VolumeType":"gp2"},"VolumesPerInstance":2}]},"InstanceGroupType":"CORE","InstanceType":"m5.4xlarge","Name":"Hadoop Workers"},{"InstanceCount":1,"InstanceGroupType":"MASTER","InstanceType":"m5.4xlarge","Name":"Hadoop Master"}]' \ --configurations '[{"Classification":"hdfs-site","Properties":{"dfs.replication":"1"}},{"Classification":"mapred-site","Properties":{"mapreduce.map.memory.mb":"3584","mapreduce.reduce.memory.mb":"8192","mapreduce.map.java.opts":"-Xmx3072m","mapreduce.reduce.java.opts":"-Xmx7168m"}},{"Classification":"yarn-site","Properties":{"yarn.nodemanager.resource.cpu-vcores":"13","yarn.nodemanager.resource.memory-mb":"52428","yarn.scheduler.maximum-allocation-mb":"52428","yarn.app.mapreduce.am.resource.mb":"2048"}}]' \ --auto-scaling-role EMR_AutoScaling_DefaultRole \ --ebs-root-volume-size 100 \ --service-role EMR_DefaultRole \ --name 'Kylin4 Benchmark' \ --scale-down-behavior TERMINATE_AT_TASK_COMPLETION \ --region cn-northwest-1
Table of Content