[simulationcraft.org] Publish non-patchwerk reports

[navv1234]

Originally reported on Google Code with ID 2181

I really can't count the numbers I saw discussions about simc being wrong, or only being
singletarget, etc. .. You have seen them as well.

Can we please just add another fight_style to the website? I'd vote for HecticAddCleave
but others could work fine as well. As long as people see different stack ranks and
finally understand that spec performance depends on fight types, lengths, number of
adds, type of movement, etc.

Reported by dr.max.moellers on 2014-10-29 14:54:19

[navv1234]

That could be worse, as some specs have really good action lists for aoe/movement, and
some don't.

Reported by s.alexlowery on 2014-10-29 15:39:29

[navv1234]

I gladly spend a couple hours to improve APLs if that means I do not have listen to
those badly-informed payers / blizzard representatives anymore that claim simc is stupid
and should be generally ignored.

Reported by dr.max.moellers on 2014-10-29 16:42:33

[navv1234]

I'm agree, and I'm all for it... just saying that it's going to take some time to get
every spec up to speed. At minimum I'll have to port some version of warrior movement
to all melee, as I have range checks for all abilities instead of halting everything
during a small movement event. 

Fury's action list is a good example of something that works well with raid events.

Reported by s.alexlowery on 2014-10-29 17:36:01

[navv1234]

APLs that presumably need work:
http://tinyurl.com/n8p43rx Bottom half performing specs from a 5 target fight. The
following specs are worth taking another look at for AOE performance:

Monk_WW_2h, Monk_BM
Priest_Shadow
Hunter_SV, Hunter_MM
Rogue_Assi
Warlock_Destro, Warlock_Demo
Warrior_Prot (although that one might be fine)
DK_Blood
Mage_Arcane
Druid_Guardian
Paladin_Prot.

http://tinyurl.com/mf4zycz bottom half performing specs for heavymovement.
The following specs are worth taking another look at for MOVEMENT:

sham_elemental
DK_frost
WL_affli,WL_Demo
Mage_Arcane, Mage_Fire
Priest_shadow
Druid_Balance
Rogue_Combat
Paladin_Prot
Warrior_Prot
DK_Blood
Druid_Guard
Monk_BM

I will double check the Warlock ones. I'd assume that most of the tank stuff is also
reasonable, but we should still take closer look.

Reported by dr.max.moellers on 2014-10-29 18:15:26

[navv1234]

The tanks are fine, tanks do no where near as much damage as they did in MoP.

Reported by s.alexlowery on 2014-10-29 18:44:06

[navv1234]

I thought they where aiming at 75% of the dps.. So everything but the brewmasters looks
okay-ish.

Reported by dr.max.moellers on 2014-10-29 18:49:09

[navv1234]

WL_Destro is fine now for AoE and can't really be improved for movement.
I will contact ghaddo for demo improv.

Reported by dr.max.moellers on 2014-10-29 19:56:18

[navv1234]

Adding more reports to front page means something has to give. The server that generates
those reports is used to do continuous testing on the simulator, and I do not think
it's a good idea to sacrifice that functionality for the benefit of the masses who
will always think simc is shit, regardless of the reports.

Reported by navv1234 on 2014-10-29 20:14:58

[navv1234]

Currently we are hosting Mop 16M, WoD 16M, WoD 17M... 
Is it really much more computations to just have WoD 17m-patchwerk and WoD 17M-HecticAddCleave?

Reported by dr.max.moellers on 2014-10-29 20:18:40

[navv1234]

MoP 16M is not computed. It takes 2 hours of 200% cpu to generate those two other reports,
so each added style adds about 1 hour of computation to that job.

Reported by navv1234 on 2014-10-29 20:22:43

[navv1234]

But we do not need WoD 16M anymore in a couple of weeks, so the cycles should be free.

Reported by dr.max.moellers on 2014-10-29 20:24:20

[navv1234]

We have generally kept "previous tier" and "current tier" there. That being said, I
suppose we could add a third report to the generation, with the hope that linode does
not suspend us for abusing CPU cycles :)

Reported by navv1234 on 2014-10-29 20:26:06

[navv1234]

Yay!!!

Maybe the improved estimated of iterations can cut times so we have enough cycles left.

Reported by dr.max.moellers on 2014-10-29 20:35:42

[navv1234]

Another option could be doing patchwerk on mon/wed/fri, and hecticaddcleave on the other
days. Our raid reports do not change enough day to day to matter.

Reported by s.alexlowery on 2014-10-29 21:28:37

[navv1234]

There isn't much to do in terms of AOE action lists for Mages. Arcane is that low simply
because it has no viable sustained AOE without talent changes. To the best of my knowledge,
Blizzard believes this as "working as intended". This is also why the default AOE raid
profiles have talent and glyph changes in them.

Movement action lists aren't just "a couple hours more work", but a lot more than that.
Mage sims are not designed to cope with heavy movement, and they share the same talent
selection issues as the AOE scenario. One obvious source of problems is Rune of Power,
which would require significant improvements not only in APLs, but also in raid movement
implementation. On top of this, I do not think there is much room for added complexity
in the Mage APLs to cope with complex movement scenarios.

Ultimately, I do not believe that any sort of support for AOE/movement/non-Patchwerk
fights will actually solve this problem. From my experience, most critics won't be
persuaded by a stacked chart of non-Patchwerk sims, because their minds are often already
set on "Sims are inaccurate", regardless of evidence. This is often sentimental and
due to "personal experiences", ie. "I'm not seeing these charts in my raids".

For these critics who already have a set agenda, "Sims don't have movement/AOE" is
just one of many (unfounded) attacks that they can make. Generating charts to address
this only leads to them moving to the next attack they can think of. When they start
claiming "players don't execute rotations perfectly", are we going to generate several
charts with varying skill settings as well? When they claim "players don't all have
the same reaction time", do we also create multiple charts for each reaction time setting?
There is no end to the claims they can make to call the sims "invalid", and I don't
see addressing them one-by-one as a good solution.

Reported by kuenmao on 2014-10-30 18:54:08

[navv1234]

Hi kuenmao,

thanks for your input. I share your doubts that we will not silence every naysayer
no matter how many hours we spent. However, I still think we can address the most prominent
concerns and thereby convert a larger fraction of "disbelievers".

You are totally right that we can't put up charts for every boss fight, every skill
level, class etc. But the change from just having one list that is perceived as "the
absolute truth" on how classes are ranked to two lists is huge. Because with two lists
even stupid people will realise that there are several factors that come into play.
(something that we and some Blizzard employees) are well aware of.

If you say mages are hard to sim with movement, then we might need to look for more
commiters in that area instead of totally ignoring the fact that the module doesnt
give usable results.

Regarding talent selection and APL complexity: Wouldn't adding a couple of talent_overrides
in the precombat area be a starting point? We can always go from there.

Greets 
Max

Reported by dr.max.moellers on 2014-11-02 11:56:59

[navv1234]

After playing around with the mage arcane profile a little bit, I agree that making
some of the talents work in an add / movement heavy fight looks indeed tricky.

However, just switching to Incanters + Arcane Orb does give very good results.

More importantly, I think this talent switch does not only make the simming job easier,
I would expect most player to switch to those (easy-to-play) talents anyway, i.e.,
this talent override is not only a bandaid for our limited ressources but actually
resembles how people play the spec.

Reported by dr.max.moellers on 2014-11-02 12:16:35

[navv1234]

The only problem with talent override is if the player wants to change talents, it may
not be obvious that they need to remove that line in order to do so. I changed the
warning for override to always show up whenever a talent is overridden, but that may
not be enough.

Reported by s.alexlowery on 2014-11-02 16:02:08

[navv1234]

Thanks for responding, Max.

The issue of movement and AOE isn't just a manpower issue. The current structure of
the simulation simply does not fully support a good modelling of movement. Off the
top of my head, there are a few issues:

1. Actor positioning currently assumes a 1-dimension plane. This means that AOE and
movement and out-of-range situations cannot be modelled. There is no "looking ahead
and preparing for movement events ahead of time". There is no "repositioning the player
while staying at range". In a 1-dimensional plane, there is only moving towards the
boss, or away (in either direction).

2. There is no full-fledged targeting system, and the APL syntax also does not offer
easy-to-use tools to model rotations that involve complex target switching. The mage
module currently has a number of custom made tools made for handling Prismatic Crystal,
which are meant to "beta test" some potential additions for all actors. Even so, some
critical assumptions are made in the sim, such as requiring a target for every ability
and interrupting it when the target is dead. This means that precasting channeled AOE
(ie. Blizzard) at a location with no target is impossible under the current design.

3. Raid events such as movement or AOE add generation does not resemble reality, because
there are many types of movement in raids that cannot be described within our current
framework. There is no distinguishing between "move out of fire but stay around the
same area" and "move forward 40 yards because the boss is moving as well". A ranged
character has more than "stay within 5 yard melee range" to worry about, and the situation
gets even more complicated when you take into account character facing direction and
immobile casting. How would you model a player that finishes a cast standing in 1 tick
of fire, versus having to move immediately due to fire being lethal?

4. Lack of support in generic actors/players for coping with movement. Paraphrasing
Reia, a realistic player doesn't just meet some minimum distance/time moved and then
stay there. There is either some sort of moving back, or looking ahead for positioning.
To model this correctly, some sort of AI must be implemented for a generic actor.

5. Currently, action lists are static and do not offer the option of adjusting spell
order based on target numbers. To create a true AOE rotation without involving an impossible-to-manage
APL, dynamic APLs using DPET as an input for priority ranking would be required. This
is a shortcoming in our current APL language.

6. Last of all, movement during casting requires a whole new system, especially for
traditionally immobile casts. There is no functionality for repositioning to target
location during the next period of time.

Don't get me wrong - I'm not saying to ignore AOE or movement. AOE is in fact decently
supported by the Mage specs, as you have noted. The issue is that it doesn't resemble
realistic actions, and it only gets worse when movement is involved. The current AOE
raid profiles which offer talent overrides (which by the way, I greatly object to doing
so in precombat) are performing fine, but.

It is fine if you want to add another chart to tell the world that Patchwerk isn't
the "one-true-answer". I have no objection to that. But the fact is, there will be
a lot of criticism from intelligent players who actually look into a hecticcleave sim,
and identify that their play does not resemble reality at all. Those will be valid
criticisms, and the responsibility will lie with us for deciding to publish high profile
information that turns out to be heavily misleading.

Reported by kuenmao on 2014-11-02 16:18:49

[navv1234]

And to clarify the things I said: Yes, all of those can be solved with sufficient manpower
and time. But most of them require excessive work, and I have doubts that they will
yield enough reward to justify the developer time spent. I think that those efforts
would be better spent on improving the existing components that already require support.

Reported by kuenmao on 2014-11-02 16:21:08

[navv1234]

I disagree to putting up non-Patchwerk fight to simulationcraft.org as well.

A lot of reasons have been mentioned, so I won't repeat them all. For me, simulationcraft.org
just isn't the place to put highly experimental aoe/interrupt/movement simulations,
especially since the quality of the class modules and the APL's varies so much.
Most classes have their specialised forums or threads on general discussion board,
where it might be suitable to post and analyze whatever simulations are appropriate
for that class, with the necessary quality standards to ensure conclusions can be drawn
from the simulation.

I also like the idea of the raid reports on simulationcraft.org as some sort of 'teaser'
for the masses, bringing us attention and feedback, good and bad. But I don't think
putting more detailed/better reports up there will change much about the quality of
the responses to them. Those really interested can so easily set up SimC themselves,
do more detailed analysis and base their critique on that.

Besides the class modules implementation, core support for a lot of things required
to model more complicated fights is experimental at best. Extended movement code is
very new, raid event predictions as well and good target support is still missing.
There certainly might be a time ( next patch, next addon ) where aoe/movement simulations
can be done with a lot more confidence across the board, and when we can re-evaluate
this request.

Reported by philoptik on 2014-11-04 09:24:07

• Status changed: Accepted
• Labels added: simulationcraft.org

[navv1234]

Just want to mention that most of the arguments against aoe sims also apply to patchwerk
fights (e.g. quality of modules and APLs). Surprisingly, this seems not to be a problem
for putting up the stack ranks in this case. 
I wholeheartedly believe that it is the wrong to only publish one scenario, but I won't
repeat the arguments over and over again.

Reported by dr.max.moellers on 2014-11-04 09:53:17

[navv1234]

Most people who do default APLs spend the vast majority of their time optimizing, and
developing good single target action lists. While different modules are in different
state of maintenance, the effort spent on single target vs aoe default action lists
is significantly different in general.

Secondly, I'd also wager that AOE modeling in the class modules is not as well maintained,
or feature complete as single target modeling, largely due to the previous point.

Also, raid event predictions are available as expressions, have been for a while.

Reported by navv1234 on 2014-11-04 11:25:18

[navv1234]

I still think that publishing aoe results will have a similar push on the quality of
the aoe profiles and class modules as does publishing single target results has on
the quality of the single target profiles and class modules. 

And it would definitely make people aware of the fact that simcraft *can* support different
movement and target scenarios and they will start exploring.

Reported by dr.max.moellers on 2014-11-04 12:35:58

[navv1234]

FWIW, I agree with dr.max.moellers. There's a chicken and egg problem here. AoE support
is mediocre, because there's little interest in AoE support or limited understanding
that AoE can be supported by SimC. The single target rankings advertise and promote
the single target value and capabilities of SimC which probably helps to focus efforts
on single target improvements. Perhaps AoE rankings might do the same thing.

Also, I don't think this is a matter of 'giving in' and trying to fight off the of
anti-simulation arguments. There's an 80/20 rule here just like everything else in
life. Maybe AoE rankings provide a lot of value (highlight the flexibility of SimC,
persuade APL contributors to take AoE more seriously, etc) for minimal effort (1 more
automated sim). Or, maybe not.

Reported by xythian on 2014-11-04 17:54:11

[navv1234]

You seem more interested than many here (or, perhaps you just have the time) - the APLs
are public, feel free to begin optimizing APL rotations for AoE while not sacrificing
the fidelity of the single target rotations. It'd probably help your case if such improvements
were made.

Reported by dzahorn1 on 2014-11-08 04:52:46

[navv1234]

I've been wrestling with the issue of movement as well - as I make my stat weights for
WoD on askmrrobot.com. 

I think their might be another approach worth considering for tackling this issue of
modeling movement in the simulator. In a nutshell, do it in a much more abstract manner.

Most people want to use SimC to try to figure out what stats they should get on their
gear and/or what order they should use their abilities in. They aren't actually setting
up detailed models of specific boss fights. So, a "hecticcleave" or "hecticaddcleave"
simulation is really meant to approximate and simulate a player's overall raid experience
across multiple attempts on multiple bosses. In that context, it doesn't really make
sense to model precise movement events where you go from location a to location b,
and the boss goes from location x to location y, then calculate a distance to the boss
and check abilities in between. You would want that information to vary greatly from
iteration to iteration to find the results that people want from the simulator.

Instead, I think we can boil down "movement" events to 2 different types:
1.) You move for some amount of time and cannot hit the boss/adds while you are moving.
2.) You move for some amount of time and can hit the boss/adds with abilities that
can be used while moving.

So, for our simulations, we can add two raid events to the configuration in a form
like:
raid_events+=/movement,cooldown=20,cooldown_stddev=10,distance=10,distance_min=5,distance_max=55,distance_range=50,player_chance=0.1,ability_use=0

raid_events+=/movement,cooldown=20,cooldown_stddev=10,distance=10,distance_min=5,distance_max=55,distance_range=50,player_chance=0.25,ability_use=1

This will create a random amount of movement events on each iteration. Some where you
can use abilities, and some where you can't. We would want to discuss the frequency
and chance of each type of event, based on our raid experience, so don't hold me to
those numbers right now - they're just for illustration.

I think that this ends up modeling what we need: random breaks in the APL, which is
ultimately what movement causes in a raid. I know that there are situations (especially
for melee classes) where they might be able to hit with ranged abilities but not auto-attacks
while moving... but who can know how often that would happen? If we think it would
affect the results drastically, we could maybe add another flag that indicates if they
remain in melee range during the move, and put that in as another random variation
of the movement event.

To summarize, where you actually move doesn't matter overly much for the results most
of us are trying to obtain. What matters is how long you have to stop using your abilities.
It might be sufficient to model movement as two or three random raid events that limit
the player's ability usage appropriately for the duration.

Reported by eric@askmrrobot.com on 2014-11-09 20:42:10

[navv1234]

I think we should start by having a concurrent aoe run just to help polish the aoe elements
of specs. Having a single spec that can handle single target and aoe is pretty straightforward,
and simply having single target vs. 4 target could help reduce single-target dps focus.

In general, movement is much more complicated than aoe because there are so many more
options. For example, spec APLs may deal differently with:
- predicted movement
- predicted periods where movement is likely (so you change spell priority)
- periods where movement *soon* is necessary (e.g., run-out for dispel can wait a second
to finish a cast)

Predicting movement is actually currently supported. At least a parameter would need
to be added to support other movement modes.

Note though that a "movement" that prevents all actions can just be modeled as a "stun"
raid event, so there's little need to complicate movement events for that. 

Reported by lokrickOrc on 2014-11-30 15:53:40

[navv1234]

I'd be ok with a 4 target aoe simulation. I can make a raid profile specifically for
it, so that all classes use their aoe stuff.

Reported by s.alexlowery on 2014-12-02 11:42:17

[navv1234]

4 looks like a good number and I can see your points for not having dynamic movement
or add spawn events. I still would like to see logs for that as well, but the 4 target
is a good start :-)

Reported by dr.max.moellers on 2014-12-02 11:54:06

[navv1234]

Just note, that cycle_targets seems unoptimized, and slow simulation up to 2+ times.

Reported by antipchel on 2014-12-03 02:53:30

[navv1234]

cycle_targets adds more complexity to the target selection and a lot more dots are active
on targets, thus we have a lot more to process. We can always double check the code,
but from a first inspection I see not much we can do to improve the situation.

Reported by dr.max.moellers on 2014-12-03 09:39:29

[navv1234]

Yeah, cycle_targets is fairly optimized, there's just not much you can do when it has
to check every target for debuffs. 

Reported by s.alexlowery on 2014-12-08 19:07:04

[navv1234]

I'm going to put a aoe sim up, so we can see what it looks like. What would the exact
options be to cover a broad spectrum of valuable information?
I guess main points are: number of enemies, and simulation length.

Reported by philoptik on 2015-01-17 17:24:55

[navv1234]

Hmm must valuable would be 2-3 minutes of fight with say. 5 ? enemies and no heroism

this would give an overview of sustained AoE.

Reported by dr.max.moellers on 2015-01-17 17:57:56

[navv1234]

Ok so it looks like the aoe sim has been published to the main site. It was with 4 targets
and 300s, will be 180s tomorrow.

Reported by philoptik on 2015-01-23 20:31:23

[navv1234]

looks good to me. and apparently some specs need an update in their APLs :-) Windwalker
/ Arcane Mage are pretty obvious candidates, but I have no clue on their optimal rotation.

Also, deactivating bloodlust would be a reasonable change.

Reported by dr.max.moellers on 2015-01-25 08:44:45

[navv1234]

I think that what we have is good, but we could probably improve it even more.  I'm
going to change the 4-target sustained to 5-target, and then add in a 2-3 target cleave
encounter that will attempt to "feel" similar to iron maidens. I'm thinking of kludging
in some type of fight style that will end up doing something similar to this:

x-target encounter that does not end until all x targets are dead. Allow one of the
targets to go out of range of melee, but still be reachable by ranged, and also have
a 3rd state where the mob is not invulnerable, but is simply not targetable by new
abilities. So, things like dots will continue to tick.

I may have to finish adding in a x,y co-ordinate system to do this. I don't plan to
redo our targeting system because that's just not something I feel comfortable doing.

Reported by s.alexlowery on 2015-03-09 06:41:35

[scamille]

I think we are fine right now with the AoE reports for 6.x on simulationcraft.org

If there is a new drive for improved non-patchwerk reports in 7.0, a new ticket for improvements can be opened.