search

awslabs / deequ

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
Apache License 2.0
3.18k stars 517 forks source link

Assertion failure in anomaly detection (HoltWinters) #345

Open aldmser opened 3 years ago

aldmser commented 3 years ago

Hi, I have a problem with anomaly detection using HoltWinters analyzer. I've created small code snippet to illustrate it:

new HoltWinters(Daily, Weekly).detect(
      Vector(
        258.0,
        245.0,
        142.0,
        471.0,
        560.0,
        724.0,
        758.0,
        647.0,
        623.0,
        292.0,
        337.0,
        415.0,
        430.0,
        567.0,
        686.0,
        692.0,
        477.0,
        453.0,
        697.0,
        708.0,
        171.0
      ), (20, 21)
    )

running it causes this exception:

Exception in thread "main" java.lang.AssertionError: assertion failed
    at scala.Predef$.assert(Predef.scala:156)
    at breeze.optimize.LBFGSB.determineStepSize(LBFGSB.scala:115)
    at breeze.optimize.LBFGSB.determineStepSize(LBFGSB.scala:31)
    at breeze.optimize.FirstOrderMinimizer$$anonfun$infiniteIterations$1.apply(FirstOrderMinimizer.scala:61)
    at breeze.optimize.FirstOrderMinimizer$$anonfun$infiniteIterations$1.apply(FirstOrderMinimizer.scala:58)
    at scala.collection.Iterator$$anon$7.next(Iterator.scala:129)
    at breeze.util.IteratorImplicits$RichIterator$$anon$2.next(Implicits.scala:74)
    at breeze.util.IteratorImplicits$RichIterator.last(Implicits.scala:87)
    at breeze.optimize.FirstOrderMinimizer.minimizeAndReturnState(FirstOrderMinimizer.scala:114)
    at breeze.optimize.FirstOrderMinimizer.minimize(FirstOrderMinimizer.scala:110)
    at com.amazon.deequ.anomalydetection.seasonal.HoltWinters.modelSelectionFor(HoltWinters.scala:168)
    at com.amazon.deequ.anomalydetection.seasonal.HoltWinters.detect(HoltWinters.scala:226)

could you help me please with advice on how am I supposed to fix that?

Thanks.

zeotuan commented 4 months ago

Since you are using Daily Interval with Weekly Seasonality, This should work:

new HoltWinters(Daily, Weekly).detect(
      Vector(
           258.0, 245.0, 142.0, 471.0, 560.0, 724.0, 758.0, 
           647.0, 623.0, 292.0, 337.0, 415.0, 430.0, 567.0, 
           686.0, 692.0, 477.0, 453.0, 697.0, 708.0, 171.0
      ),
      2 * 7 -> 3 * 7
)