Open xiandong79 opened 7 years ago
hi, Xiandong
Thanks for interesting, I wrote a generator here for my paper: https://github.com/yncxcw/YarnBench
It now can (1) support trace jobs generation( from a trace file) and poisson job generation( from user configurations) (2) monitor cluster information via REST API. I do not have a document for it yet, I will do soon. You can refer conf.default as a starting point as well as some code for your own purpose.
For more questions, feel free to contact me.
Wei Chen
Thank you so much.
Q1: Will you categorize Big-C into Yarn on Docker, Docker on Yarn or a new type?
Maybe Docker on Yarn?
Q2: Does each time a new job is submitted, a new docker/container is launched?
If so, I am concerned about the cost of launching a docker frequently, 10ms per job?
Hi, Xiandong
For Q1, I would rather say Big-c is a yarn on docker. Since yarn now already supports docker containers, we leverage the docker as well as cgroup in our project.
For Q2, It depends on many factors, like your hard drive, image size ... I test one of my images(I build jdk, hadoop and spark in this image) on my server (equipped raid-5 hdd):
admin@host7:~$ time docker run -d cwei/hadoop:3.0.0 /bin/bash 0ab19bae13f7d2020a966aed6c9c76712faa73d194377941a154c8ba2fe4f65a
real 0m0.354s user 0m0.020s sys 0m0.004s
It turns out launching a container takes 0.35s which could be a big problem for a 10ms task. However, based on my experience I did not find any frameworks(mapreduce task and spark executor) that have a duration of 10ms yet, since JVM launching is still slow. The shortest task I observed was about 10 seconds.
By the way, I would love to know if you are using some tiny workloads, since I am trying to optimize the scheduling delay for YARN.
Thanks for your illustration.
I understand your idea that compared with a job costing 10 mins, 10ms launch time is negligible.
What about SparkStreaming jobs in Hibench where the task durations are relatively small?
@xiandong79
I understand. I think streaming jobs may not be fit for this situation because the spark executors are allocated at the beginning of the job and the resources will also be kept during its lifetime.
Wei
Hi, This is Xiandong(显东)。
Would you provide the ''job submitting'' scripts of your original experiments? I am preparing a similar project but have no idea how to design a "load generator" as described in your paper.
My Email: xqiad@connect.ust.hk, Wechat: qi839395901.