search

simulationcraft / simc

Simulationcraft engine/GUI
GNU General Public License v3.0
1.39k stars 693 forks source link

smooth_rng changes DPS #281

Closed navv1234 closed 9 years ago

navv1234 commented 9 years ago

Originally reported on Google Code with ID 282

Depending on the smooth_rng setting the simulation converges to differing DPS values.

With the files given and the output given, the only setting difference was smooth_rng
(10^6 
iterations), yet the DPS changed from
   9542.1 (+/- 0.5, range=1070)  Hundinglegs
   9531.0 (+/- 0.5, range=1044) Hundinggloves
   9485.2 (+/- 0.5, range=1079)  Hundingbase

to 

DPS Ranking:
   9580.6 (+/- 0.2, range=418)  Hundinglegs
   9571.0 (+/- 0.2, range=422) Hundinggloves
   9523.4 (+/- 0.2, range=404)  Hundingbase

These differences are outside the margin of error given (0.5 and 0.2)

Additionally, the range changed significantly (1079 to 404 for Hundingbase) as did

elemental_focus

If scale factors were being generated from these it would have an impact, as the DPS
values do 
not differ by a constant multiplier (i.e. expressed as percentages of Hundingbase they
change), 
leading to errors that will not be removed by normalization against a common stat (e.g.
spell 
power)

Reported by penorman on 2010-03-25 07:18:42

navv1234 commented 9 years ago 
Based upon your output (nicely detailed, thank you) it would seem that everything is
working as intended.... but perhaps not as advertised/documented.

The smooth-rng code is black-magic designed to tighten up the DPS bell-curve.  Note
that the range and error distribution for smooth-rng runs are smaller than standard
runs.  Comparing two smooth-rng runs against each other for the purposes of scale
factor generation will reduce the inherent "noise" in such calculations.  

I would not recommend comparing a smooth-rng run to a non-smooth-rng run.  While
there is reasonable fidelity between comparing on-to-on and off-to-off, the
smooth-rng run will almost always have a slightly higher dps value.  

The purpose of smooth-rng is to enable scale factor generation when runtime is
constrained.  Simulations with smooth-rng will converge 10x to 100x times faster than
standard runs.  In your example this means that that your smooth-rng run could
possibly done with only 10^4 iterations.

The trouble is that for some specs of some classes, discontinuities appeared in the
DPS curves when smooth-rng is enabled.  It is not an endemic problem, but frequent
enough that we could not turn it on by default....  Indeed, we throw it up on the
options list but carefully make little mention of it.  :-)

Personally, I reserve smooth-rng for generating DPS plots.  With smooth-rng I can set
the iterations to only 100 and get reasonable looking DPS plots very quickly.

I wish I could smooth out those discontinuities, but I've already spent hours and
hours and hours down that rabbit hole.  So I am going to label this Issue as "Wont
Fix".  If you do some more analysis and provide some new insight/intuition, perhaps
we can reopen it once more.

With kind regards,
Nathaniel

Reported by natehieter on 2010-03-26 15:23:00

navv1234 commented 9 years ago 
Do you have a reference for the normalized rng via variable phase shift (presuming that's
the one used)?

I was looking at Numerical Recipes in C ('92 edition) for any references, and I was
wondering if there might be 
an issue with a correlation between subsequent random numbers (p277)

"It has the disadvantage that it is not free of sequential correlation on successive
calls. If k random numbers 
at a time are used to plot points in k dimension space (with each coordinate between
0 and 1), then the points 
will not tend to "fill up" the k-dimensional space, but rather will lie on (k-1) dimensional
"planes". ..." (on 
linear congruential methods)

Reported by penorman on 2010-03-27 03:34:04

navv1234 commented 9 years ago 
Rolls that occur "often" use the phase-shift RNG.   
Rolls that occur "occasionally" or have a very strong effect use the distribution RNG.

phase-shift: fast, but periodic spacing
distribution: slower, but has variable (still deterministic) spacing

If you want to tackle this send me a Google ID and I'll happily grant you commit access!

Reported by natehieter on 2010-03-27 13:35:21

navv1234 commented 9 years ago 
google ID is this ID @ mac dot com

I can't figure out how to get your contact info directly

Reported by penorman@mac.com on 2010-05-02 21:15:27