Overflow when running random, bayesian CP detectors (in scipy)

[breznak]

I've installed this new NAB and test by running all default detectors.

I'm observing so random overflows logged:

25: Results have been written to /mnt/store/devel/HTM/NAB/results/null/realAWSCloudwatch/null_ec2_cpu_utilization_fe7f93.csv
. 143: Beginning detection with bayesChangePt for realAWSCloudwatch/ec2_disk_write_bytes_c0d644.csv
. . . . . /mnt/store/devel/HTM/pyenv3/lib/python3.6/site-packages/scipy/stats/_continuous_distns.py:5750: RuntimeWarning: overflow encountered in power
Px /= np.sqrt(r*np.pi)*(1+(x**2)/r)**((r+1)/2)
. . . 72: Completed processing 4032 records at 2020-02-05 16:33:30.631350
72: Results have been written to /mnt/store/devel/HTM/NAB/results/random/artificialWithAnomaly/random_art_daily_jumpsup.csv
73: Beginning detection with random for artificialWithAnomaly/art_daily_nojump.csv
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 73: Completed processing 4032 records at 2020-02-05 16:33:33.141159
73: Results have been written to /mnt/store/devel/HTM/NAB/results/random/artificialWithAnomaly/random_art_daily_nojump.csv
74: Beginning detection with random for artificialWithAnomaly/art_increase_spike_density.csv
. . . . . . . . . . . . . . . . . . . . /mnt/store/devel/HTM/pyenv3/lib/python3.6/site-packages/scipy/stats/_continuous_distns.py:5750: RuntimeWarning: overflow encountered in multiply
Px /= np.sqrt(r*np.pi)*(1+(x**2)/r)**((r+1)/2)

Eg. this:

74: Beginning detection with random for artificialWithAnomaly/art_increase_spike_density.csv . . . . . . . . . . . . . . . . . . . . /mnt/store/devel/HTM/pyenv3/lib/python3.6/site-packages/scipy/stats/_continuous_distns.py:5750: RuntimeWarning: overflow encountered in multiply Px /= np.sqrt(rnp.pi)(1+(x2)/r)((r+1)/2)

[rhyolight]

@breznak What environment are you running this in?

[breznak]

I'm on current master, ubuntu 18.04, python3.6,

installed with

pip install . --extra-index-url https://test.pypi.org/simple/

run command:

python run.py --detect --optimize --score --normalize -n 8

pip list:

Package            Version   
------------------ ----------
atomicwrites       1.3.0
attrs              19.3.0
boto3              1.9.134
botocore           1.12.253
certifi            2019.11.28
chardet            3.0.4
cycler             0.10.0
Cython             0.29.14
docutils           0.15.2
Hexy               1.4.3
htm.core           2.0.22
idna               2.8
importlib-metadata 1.5.0
jmespath           0.9.4
joblib             0.14.1
kiwisolver         1.1.0
matplotlib         3.1.3
mock               4.0.0b1
more-itertools     8.2.0
nab                1.1
numpy              1.18.1
packaging          20.1
pandas             0.23.3
pip                20.0.2
pkg-resources      0.0.0
plotly             2.0.0
pluggy             0.13.1
prettytable        0.7.2
py                 1.8.1
pyparsing          2.4.6
PyQt5              5.14.1
PyQt5-sip          12.7.1
pytest             4.6.5
python-dateutil    2.8.1
pytz               2019.3
requests           2.22.0
retrying           1.3.3
s3transfer         0.2.1
scikit-learn       0.21.1
scipy              1.4.1
setuptools         39.0.1
simplejson         3.11.1
six                1.14.0
urllib3            1.25.8
wcwidth            0.1.8
wheel              0.34.2
zipp               2.1.0

[rhyolight]

I am going to start work on this today in the appropriate docker setup.

[rhyolight]

I have replicated this error in Ubuntu 18.04 in both Python 3.6 and 3.7. Unfortunately, I only tested this on macOS in Anaconda and Python 3.7. Standby I need to talk to the team about how to fix this.

[breznak]

Unfortunately, I only tested this on macOS

if it works in OSX, seems it's not in our code per se but some numeric instability in the computations. Might help

• try updating scpipy, numpy?

[rhyolight]

It is a scipy dependency problem in the Ubuntu environment. I set up bad env just like you in docker. Here is a docker command that will work:

docker run -it -v `pwd`:/nab python:3.7 /bin/bash

Then run:

cd nab
pip install -r requirements.txt
pip install .
python run.py --detect --optimize --score --normalize -n 8

So this works in 3.7. I am testing 3.6.

[breznak]

It is a scipy dependency problem in the Ubuntu environment. I set up bad env just like you in docker.

have you been able to pin-point the problem to which version, of scipy or its dependency it is in?

[breznak]

Seems random runs just fine, so only in Bayes. This is a minimal reproducer for me: python run.py -d bayesChangePt --detect --optimize --score --normalize -n 8

I do not think the err is solved by running Docker env. (Maybe the warning is just silenced there?)

It seems in NAB we should set proper dtype, https://stackoverflow.com/questions/7559595/python-runtimewarning-overflow-encountered-in-long-scalars

[rhyolight]

I don't know which scipy version is the problem. Why don't you think the error is solved by running in Docker? It ran find and produced results.

[rhyolight]

Because I could only replicate this with a bad python environment, I'm going to assume this is a python environment problem. It works fine with the python3.7 docker image.

[breznak]

Why don't you think the error is solved by running in Docker? It ran find and produced results.

not a priority, but I thought only py2 detectors have to run in Docker, others should run on normal linux/osx.

[rhyolight]

Not sure what you mean. Docker is normal linux. You just have something wrong with your python dependencies. The fact that I can run a docker command with the defacto standard python3.6 & python3.7 builds and run NAB shows this.