search

nchammas / flintrock

A command-line tool for launching Apache Spark clusters.
Apache License 2.0
638 stars 116 forks source link

Add datetime in the output? #133

Closed sylvinus closed 8 years ago

sylvinus commented 8 years ago

One of Flintrock's strenghts is definitely its speed, so wouldn't it make sense to show the datetime before each output line, to be able to see how much time each operation took and maybe give hints on how to optimize it further?

nchammas commented 8 years ago

This is a good suggestion.

Flintrock today already shows the total time any given operation took. A breakdown for the individual "stages", so to speak, would definitely be interesting for me and for contributors like me who are obsessed with speed.

Perhaps stats like median and max time taken for tasks in each stage (e.g. configuring ephemeral storage; installing Spark; etc.) can be shown in some minimal and unobtrusive way.

I think this fits best when it is implemented as part of #27. I would prefer not to add more to the already busy output, e.g. during launches.

Does that sound good to you?

sylvinus commented 8 years ago

Yes! I agree it's most interesting after the launch for analysis, not necessarily while it's being done.

nchammas commented 8 years ago

Just realized that this request is already covered as part of #47. Closing this in favor of that issue.