Chaos Function - Githubissues

lloesche commented 9 years ago

Can we get a cool Chaos Button? (cool meaning red, animated and with sound)

When pressed for more than 5 seconds it will cause chaos by killing some tasks from apps with more than one task.

pierluigi commented 9 years ago

@aquamatthias is there an endpoint for this? If not, shouldn't this be? Or do you envision the frontend to simply kills random tasks?

aquamatthias commented 9 years ago

We have kill functionality - should be enough?

pierluigi commented 9 years ago

out

cc @ashenden @aatumanova can you lend me your :eyes: on this one, please?

(Updated gif with no red button and showing the disabled Start Chaos button until the word CHAOS is entered)

pierluigi commented 9 years ago

@aquamatthias is the provided "warning" text clear enough in the dialog?

aquamatthias commented 9 years ago

I think it is very clear. Perhaps we should add some easter eggs if you type xyz instead of chaos. (xyz = outage, nasenbär etc)

pierluigi commented 9 years ago

We should probably also play this in the background https://www.youtube.com/watch?v=vxNnUU3eRD0

aldipower commented 9 years ago

:+1: https://developer.mozilla.org/de/docs/Web/HTML/Using_HTML5_audio_and_video

lloesche commented 9 years ago

+1 for playing the explosion sound. But please set a cookie so it's only played once per day or something like that. Otherwise if I have to run tests all day long and kill instances it might get annoying quickly.

On Wed, Jul 29, 2015 at 3:23 PM, Pierluigi Cau notifications@github.com wrote:

We should probably also play this in the background https://www.youtube.com/watch?v=vxNnUU3eRD0

— Reply to this email directly or view it on GitHub https://github.com/mesosphere/marathon/issues/1248#issuecomment-125950647 .

lloesche commented 9 years ago

Also: cool Chaos Button? (cool meaning red, animated and with sound) Was thinking more like http://www.shutterstock.com/pic-40404709/stock-photo-chaos-button.html with help from design team. Maybe get the entire browser window to shortly fade into a bright flash/white when the explosion is set of.

aldipower commented 9 years ago

LOL

pierluigi commented 9 years ago

@lloesche I wasn't actually being serious about the explosion ... :smile: but we could combine @aquamatthias's idea to have an easter egg there so that we play the sound when you type BOOM instead of chaos or something like that...

regarding the "coolness" we don't want to go overboard with this, and we need a go from the design team anyway :wink:

lloesche commented 9 years ago

@pierlo-upitup I would love for this to be a completely bonkers over engineered incredibly well polished function that will surprise customers. A plain button is boring. If I cause my 50k node datacenter to explode it damn well should feel like I just pushed the button to have the world go under. Imho.

Remember that presentation where our customers are supposed to feel like superstar racing car space pilots? Dan the datacenter operator does not feel like a space pilot if he operates a slider and clicks a green button. Just saying.

lloesche commented 9 years ago

btw. Instead of having to type CHAOS it would also be cooler if I could just keep the button held down for like 5sec and it would slowly fade from like green to red or some other progress indicator, before setting of CHAOS :smile: I mean the ultimate coolness factor would be if two users had to simultaneously turn a virtual key, but I guess that might be a bit overdone.

ashenden commented 9 years ago

@pierlo-upitup who will this button be shown too? Assuming this is just for demo purposes? If so, we should tuck it away so it's not something the everyday user has visibility too.

The button behaviour you added to the modal is great. Disabling the button until all required fields have been set is a UX convention we should make a habit of carrying across every form. The button though should be red, given the destructive impact it has on your system.

lloesche commented 9 years ago

@ashenden no this would be a generally available function. It's the WebUI equivalent of what dcos chaos is on the CLI.

The purpose of this is testing the failure tolerance of your applications. It's a thing that Netflix originally started with their Chaos Monkey: https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey

ashenden commented 9 years ago

What is the use-case frequency of this action though. It doesn't seem like a common function that warrants equal treatment alongside the "New App" button -- and should instead be nested away behind a few clicks.

lloesche commented 9 years ago

Absolutely agree. It's something I would use whenever a new major app release was being deployed to test it's resilience. Once every few weeks or so.

sepiroth887 commented 9 years ago

I like the idea but please please provide an option to turn it off.

If this is introducing random failure in a prod environment and its not as reliable as netflix im not sure id like to have a disgrutled person hit it.

I think we all know how great lots of ditributed stores are thanks to jepsen tests.

aldipower commented 9 years ago

I am still unsure where we should place the button? @ashenden

ashenden commented 9 years ago

@aldipower we should not make this action visible as an up-front button. Consider placing this in a dropdown next to "New App" with a gear or "More..." label -- but we should try to find at least one additional line-item to place in that dropdown.

cc/ @cleverroffey and @aatumanova

aatumanova commented 9 years ago

@aldipower @lloesche let's step back and think about the design of this button a bit more! Some questions below:

How does a user stop chaos? Can they return to the state of the world pre-chaos once they have simulated chaos? If things are going badly during the chaos, we don't want to leave our user helpless in the sidelines, watching everything get destroyed! Do we allow the user to specify a set amount of time they want to run the chaos? Or have a "stop chaos" action available after chaos ensues?
How can we design this so that chaos can be selectively applied? This feature should not be available across all apps — that could be pretty catastrophic and disruptive. And I am sure there are certain business sensitive apps that cannot be tested unless well planned out. There should be the option to mark these kind of apps "off the grid" for testing.

As far as where this button should live, could this be part of a larger collection of testing tools? Or be a lever in a larger "testing mode" functionality?

aldipower commented 9 years ago

@aatumanova @lloesche @ashenden Thinking about the chaos function a little bit more and having @aatumanova last sentence in mind, I have the feeling that a chaos function in the UI is the wrong place at all. Doing/triggering chaos should be a functionality available for DevOps on the command line. Which is already the case, we have such a tool.. Of course you can see and watch the impact of the chaos in the UI, because it's all the same data base. But I really think we should not expose a chaos starting function in the UI.

A chaos button is cool. But we should first be cautious and then be cool.

pierluigi commented 9 years ago

I have to agree with @aldipower . Let's remember to verify our ideas before implementing them ;)

pierluigi commented 9 years ago

Should we close this issue and get back to this at a later point? It seems to me that 1) we lack an agreement on the feature itself and 2) there currently is no right place in the UI for the Chaos button.

cc @aldipower @air @aatumanova @philipnrmn @aquamatthias

aldipower commented 9 years ago

We can put it in the IceBox. :)

aatumanova commented 9 years ago

Agree, let's return to this idea at another time.

philipnrmn commented 9 years ago

Into the icebox it goes.

jeschkies commented 7 years ago

Closing since there was no response or action for 1 1/2 years.

mesosphere / marathon

Chaos Function #1248