Unittest failure

[vgangireddyin]

Hi,

I noticed that test_nn.py is failing while I am trying to understand the analysis part.

Please find more details below.

vagrant@ubuntu-bionic:~/ottertune/server$ python3 -m unittest analysis/tests/test_nn.py -v
2020-03-13 16:09:58.846166: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
test_nn_ypreds (analysis.tests.test_nn.TestNN) ... FAIL
test_nn_yrecommend (analysis.tests.test_nn.TestNN) ... FAIL

======================================================================
FAIL: test_nn_ypreds (analysis.tests.test_nn.TestNN)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vagrant/ottertune/server/analysis/tests/test_nn.py", line 38, in test_nn_ypreds
    self.assertEqual(ypreds_round, expected_ypreds)
AssertionError: Lists differ: ['21.676', '22.829', '23.329', '27.713', '26.309', '24.269'] != ['21.279', '22.668', '23.115', '27.228', '25.892', '23.967']

First differing element 0:
'21.676'
'21.279'

- ['21.676', '22.829', '23.329', '27.713', '26.309', '24.269']
+ ['21.279', '22.668', '23.115', '27.228', '25.892', '23.967']

======================================================================
FAIL: test_nn_yrecommend (analysis.tests.test_nn.TestNN)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vagrant/ottertune/server/analysis/tests/test_nn.py", line 43, in test_nn_yrecommend
    self.assertEqual(recommends_round, expected_recommends)
AssertionError: Lists differ: ['21.676', '21.676', '21.676', '21.676', '21.676', '21.676'] != ['21.279', '21.279', '21.279', '21.279', '21.279', '21.279']

First differing element 0:
'21.676'
'21.279'

- ['21.676', '21.676', '21.676', '21.676', '21.676', '21.676']
?      ^ ^       ^ ^       ^ ^       ^ ^       ^ ^       ^ ^

+ ['21.279', '21.279', '21.279', '21.279', '21.279', '21.279']
?      ^ ^       ^ ^       ^ ^       ^ ^       ^ ^       ^ ^

There was a warning message from Tensorflow package about the CPU. Please find my cpu details below.

vagrant@ubuntu-bionic:~/ottertune/server$ lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  1
Core(s) per socket:  8
Socket(s):           1
NUMA node(s):        1
Vendor ID:           AuthenticAMD
CPU family:          23
Model:               1
Model name:          AMD Ryzen 5 1600 Six-Core Processor
Stepping:            1
CPU MHz:             3199.998
BogoMIPS:            6399.99
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           32K
L1i cache:           64K
L2 cache:            512K
L3 cache:            16384K
NUMA node0 CPU(s):   0-7
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch ssbd vmmcall fsgsbase avx2 rdseed clflushopt arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif

Thanks, Venu

[dvanaken]

The CPU warning is not related to the error, it's just letting you know that Tensorflow might not be running as fast as it could be if you were to compile it from source to use the additional instructions available on your CPU.

What version of TensorFlow are you running? If it's not version 1.12.2 (see requirements.txt) then this is most likely what's causing the error and it's actually not an error... the issue is that distinct versions of TensorFlow output slightly different values in the NN unit tests and we haven't had the bandwidth to update them to work for different versions of TensorFlow yet.

[vgangireddyin]

Thanks for the reply. I am closing the issue then.