CoordinateEncoderTest.testEncodeRelativePositionsAndRadii test failure

oxtopus commented 9 years ago

I'm encountering the following test failure on my mac laptop:

$ py.test tests/unit/nupic/encoders/coordinate_test.py -k testEncodeRelativePositionsAndRadii
==================================================================================================================================================== test session starts =====================================================================================================================================================
platform darwin -- Python 2.7.6 -- pytest-2.4.2
plugins: cov, xdist
collected 16 items 

tests/unit/nupic/encoders/coordinate_test.py F

========================================================================================================================================================== FAILURES ==========================================================================================================================================================
_________________________________________________________________________________________________________________________________ CoordinateEncoderTest.testEncodeRelativePositionsAndRadii __________________________________________________________________________________________________________________________________

self = <tests.unit.nupic.encoders.coordinate_test.CoordinateEncoderTest testMethod=testEncodeRelativePositionsAndRadii>

    def testEncodeRelativePositionsAndRadii(self):
      # As radius increases and positions change, the overlap should decrease
      overlaps = overlapsForRelativeAreas(999, 25, np.array([100, 200]), 5,
                                          dPosition=np.array([1, 1]),
                                          dRadius=1,
                                          num=5)
>     self.assertDecreasingOverlaps(overlaps)

tests/unit/nupic/encoders/coordinate_test.py:208: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <tests.unit.nupic.encoders.coordinate_test.CoordinateEncoderTest testMethod=testEncodeRelativePositionsAndRadii>, overlaps = array([ 0.64,  0.44,  0.2 ,  0.2 ,  0.16])

    def assertDecreasingOverlaps(self, overlaps):
>     self.assertEqual((np.diff(overlaps) >= 0).sum(), 0)
E     AssertionError: 1 != 0

tests/unit/nupic/encoders/coordinate_test.py:267: AssertionError
------------------------------------------------------------------------------------------------------------------------------------------------------ Captured stderr -------------------------------------------------------------------------------------------------------------------------------------------------------
/Users/amarshall/nta/nupic/tests/unit/nupic/encoders/coordinate_test.py:330: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  newPosition = initPosition if dPosition == None else (
=============================================================================================================================== 15 tests deselected by '-ktestEncodeRelativePositionsAndRadii' ===============================================================================================================================
========================================================================================================================================== 1 failed, 15 deselected in 0.14 seconds ===========================================================================================================================================

nupic state:

commit 9a07203cdf6937cdbbf5a1a5453f5a95d7989307
Merge: b6d6158 712073b
Author: Scott Purdy <scott@fer.io>
Date:   Thu Aug 20 16:06:46 2015 -0700

    Merge pull request #2468 from tomsilver/SHACHANGE

    updating nupic_modules with latest sha

nupic.core state:

commit 948fc59d9db8f02ddff2d36e45b2db4bbbd7c968
Merge: e828c9c b5bf354
Author: Scott Purdy <scott@fer.io>
Date:   Thu Aug 20 11:46:58 2015 -0700

    Merge pull request #561 from scottpurdy/master

    Remove unused import

rhyolight commented 9 years ago

@oxtopus This test ran and passed for me locally for head of master on my OS X.

breznak commented 9 years ago

yep, I encounter it on linux too

jcasner commented 9 years ago

From talking to @chetan51 it seems like this may be an issue with Random.cpp not generating values in a deterministic format. @rhyolight this is impacting @oxtopus on Mac OS X and our Jenkins server (Centos 6.5). However, neither Chetan nor I can reproduce on our Mac OS X machines. Both Jenkins and Austin get one set of overlap values and both Chetan and I get a different set (see below). This suggests that perhaps there's somehow a versioning issue in nupic.bindings / nupic.core / Random.cpp??

I added a print statement locally to get the overlaps and see:

~/nta/nupic 
$ tests/unit/nupic/encoders/coordinate_test.py
....[ 0.6   0.44  0.32  0.28  0.2 ]
............
----------------------------------------------------------------------
Ran 16 tests in 3.819s

OK
4.00s user 0.09s system 99% cpu 4.125s total

chetan51 commented 9 years ago

We should try printing the inputs to the Random class in coordinate.py, and comparing the values on different machines. If the inputs are identical, then we can compare the outputs of Random in coordinate.py, and thus isolate the issue to the Random class.

breznak commented 9 years ago

Ok, here's my output from an affected machine: https://gist.github.com/breznak/c266dd48ec093e5a5fd6

Obtained from:

diff --git a/nupic/encoders/coordinate.py b/nupic/encoders/coordinate.py
index dcf58ca..d7274a8 100644
--- a/nupic/encoders/coordinate.py
+++ b/nupic/encoders/coordinate.py
@@ -168,7 +168,9 @@ class CoordinateEncoder(Encoder):
     """
     seed = cls._hashCoordinate(coordinate)
     rng = Random(seed)
-    return rng.getReal64()
+    r = rng.getReal64()
+    print "_orderForCoordinate: IN=",coordinate," OUT=",r
+    return r

   @classmethod
@@ -182,7 +184,9 @@ class CoordinateEncoder(Encoder):
     """
     seed = cls._hashCoordinate(coordinate)
     rng = Random(seed)
-    return rng.getUInt32(n)
+    r = rng.getUInt32(n)
+    print "_bitForCoordinate: IN=",coordinate," OUT=",r
+    return r

And I ran only the following tests (deleted the others):

$ python tests/unit/nupic/encoders/coordinate_test.py > /tmp/out.txt
tests/unit/nupic/encoders/coordinate_test.py:105: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  newPosition = initPosition if dPosition == None else (
FF
======================================================================
FAIL: testEncodeRelativePositionsAndRadii (__main__.CoordinateEncoderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/unit/nupic/encoders/coordinate_test.py", line 65, in testEncodeRelativePositionsAndRadii
    self.assertDecreasingOverlaps(overlaps)
  File "tests/unit/nupic/encoders/coordinate_test.py", line 70, in assertDecreasingOverlaps
    self.assertEqual((np.diff(overlaps) >= 0).sum(), 0)
AssertionError: 1 != 0

======================================================================
FAIL: testEncodeRelativeRadii (__main__.CoordinateEncoderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests/unit/nupic/encoders/coordinate_test.py", line 50, in testEncodeRelativeRadii
    self.assertDecreasingOverlaps(overlaps)
  File "tests/unit/nupic/encoders/coordinate_test.py", line 70, in assertDecreasingOverlaps
    self.assertEqual((np.diff(overlaps) >= 0).sum(), 0)
AssertionError: 1 != 0

----------------------------------------------------------------------
Ran 2 tests in 0.280s

rhyolight commented 9 years ago

We've had some problems since introducing nupic.bindings with different errors coming up when some people install from the binary nupic.bindings wheel file on pypi. This doesn't happen if you build from source code.

Could be our build methods affecting this? Someone could try to see if there is a difference when building from source vs just running pip install nupic.

jcasner commented 9 years ago

@rhyolight the Jenkins build is using nupic.bindings built from source; it's on Centos so it cannot use the wheel file on pypi.

rhyolight commented 9 years ago

@jcasner That's actually kindof a relief.

breznak commented 9 years ago

Could be our build methods affecting this? Someone could try to see if there is a difference when building from source vs just running pip install nupic.

I've tried both binary and source, same err.

chetan51 commented 9 years ago

Thanks @breznak. Here is the output from my (unaffected) machine:

https://gist.github.com/chetan51/81a5849081b260c11b24

coordinate_test.py:102: FutureWarning: comparison to None will result in an elementwise object comparison in the future. newPosition = initPosition if dPosition == None else (

..

Ran 2 tests in 0.281s

OK

jcasner commented 9 years ago

@chetan51 @breznak @oxtopus because of part 2 of #2264 there are other tests failing that I wasn't even aware were failing. Note the failures in TestNupicRandom. Thanks for catching this @subutai! Full output is available here:

=========================================================================== FAILURES ===========================================================================
__________________________________________________ CoordinateEncoderTest.testEncodeRelativePositionsAndRadii ___________________________________________________
self = <tests.unit.nupic.encoders.coordinate_test.CoordinateEncoderTest testMethod=testEncodeRelativePositionsAndRadii>

    def testEncodeRelativePositionsAndRadii(self):
      # As radius increases and positions change, the overlap should decrease
      overlaps = overlapsForRelativeAreas(999, 25, np.array([100, 200]), 5,
                                          dPosition=np.array([1, 1]),
                                          dRadius=1,
                                          num=5)
>     self.assertDecreasingOverlaps(overlaps)

tests/unit/nupic/encoders/coordinate_test.py:208:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tests.unit.nupic.encoders.coordinate_test.CoordinateEncoderTest testMethod=testEncodeRelativePositionsAndRadii>
overlaps = array([ 0.64,  0.44,  0.2 ,  0.2 ,  0.16])

    def assertDecreasingOverlaps(self, overlaps):
>     self.assertEqual((np.diff(overlaps) >= 0).sum(), 0)
E     AssertionError: 1 != 0

tests/unit/nupic/encoders/coordinate_test.py:267: AssertionError
----------------------------------------------------------------------- Captured stderr ------------------------------------------------------------------------
/opt/numenta/jenkins/workspace/nupic-product-pipeline/build-329/nupic/tests/unit/nupic/encoders/coordinate_test.py:330: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  newPosition = initPosition if dPosition == None else (
________________________________________________________ CoordinateEncoderTest.testEncodeRelativeRadii _________________________________________________________
self = <tests.unit.nupic.encoders.coordinate_test.CoordinateEncoderTest testMethod=testEncodeRelativeRadii>

    def testEncodeRelativeRadii(self):
      # As radius increases, the overlap should decrease
      overlaps = overlapsForRelativeAreas(999, 25, np.array([100, 200]), 5,
                                          dRadius=2,
                                          num=5)
>     self.assertDecreasingOverlaps(overlaps)

tests/unit/nupic/encoders/coordinate_test.py:193:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tests.unit.nupic.encoders.coordinate_test.CoordinateEncoderTest testMethod=testEncodeRelativeRadii>
overlaps = array([ 0.52,  0.2 ,  0.16,  0.16,  0.08])

    def assertDecreasingOverlaps(self, overlaps):
>     self.assertEqual((np.diff(overlaps) >= 0).sum(), 0)
E     AssertionError: 1 != 0

tests/unit/nupic/encoders/coordinate_test.py:267: AssertionError
__________________________________________________________________ TestNupicRandom.testSample __________________________________________________________________
self = <tests.unit.nupic.math.nupic_random_test.TestNupicRandom testMethod=testSample>

    def testSample(self):
      r = Random(42)
      population = numpy.array([1, 2, 3, 4], dtype="uint32")
      choices = numpy.zeros([2], dtype="uint32")

      r.sample(population, choices)

>     self.assertEqual(choices[0], 2)
E     AssertionError: 1 != 2

tests/unit/nupic/math/nupic_random_test.py:76: AssertionError
_________________________________________________________________ TestNupicRandom.testShuffle __________________________________________________________________
self = <tests.unit.nupic.math.nupic_random_test.TestNupicRandom testMethod=testShuffle>

    def testShuffle(self):
      r = Random(42)
      arr = numpy.array([1, 2, 3, 4], dtype="uint32")

      r.shuffle(arr)

>     self.assertEqual(arr[0], 3)
E     AssertionError: 1 != 3

tests/unit/nupic/math/nupic_random_test.py:165: AssertionError
______________________________________________________________ SpatialPoolerTest.testExactOutput _______________________________________________________________
self = <tests.unit.nupic.research.spatial_pooler_unit_test.SpatialPoolerTest testMethod=testExactOutput>

    def testExactOutput(self):
      """
        Given a specific input and initialization params the SP should return this
        exact output.

        Previously output varied between platforms (OSX/Linux etc)
        """

      expectedOutput = [32, 223, 295, 307, 336, 381, 385, 428, 498, 543, 624,
                        672, 687, 731, 733, 751, 760, 790, 791, 797, 860, 955,
                        1024, 1037, 1184, 1303, 1347, 1454, 1475, 1483, 1494,
                        1497, 1580, 1671, 1701, 1774, 1787, 1830, 1868, 1878]

      sp = SpatialPooler(
        inputDimensions = [1,188],
        columnDimensions = [2048, 1],
        potentialRadius = 94,
        potentialPct = 0.5,
        globalInhibition = 1,
        localAreaDensity = -1.0,
        numActiveColumnsPerInhArea = 40.0,
        stimulusThreshold = 0,
        synPermInactiveDec = 0.01,
        synPermActiveInc = 0.1,
        synPermConnected = 0.1,
        minPctOverlapDutyCycle=0.001,
        minPctActiveDutyCycle=0.001,
        dutyCyclePeriod = 1000,
        maxBoost = 10.0,
        seed = 1956,
        spVerbosity = 0

      )

      inputVector = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0,
                     0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
                     0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
                     0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0,
                     1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
                     0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
                     0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
                     0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
                     0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
                     0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
                     0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
                     0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0,
                     1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0,
                     1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0,
                     0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
                     0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

      inputArray = numpy.array(inputVector).astype(realDType)

      activeArray = numpy.zeros(2048)

      sp.compute(inputArray, 1, activeArray)

      # Get only the active column indices
      spOutput = [i for i, v in enumerate(activeArray) if v != 0]
>     self.assertEqual(spOutput, expectedOutput)
E     AssertionError: Lists differ: [57, 80, 135, 215, 280, 281, 3... != [32, 223, 295, 307, 336, 381, ...
E
E     First differing element 0:
E     57
E     32
E
E     Diff is 751 characters long. Set self.maxDiff to None to see it.

tests/unit/nupic/research/spatial_pooler_unit_test.py:209: AssertionError
----------------------------------------------------------------------- Captured stdout ------------------------------------------------------------------------
New seed generated as: 55243 called by run line 322 -> setUp
___________________________________________________________ TemporalMemoryTest.testBestMatchingCell ____________________________________________________________
self = <tests.unit.nupic.research.temporal_memory_test.TemporalMemoryTest testMethod=testBestMatchingCell>

    def testBestMatchingCell(self):
      tm = TemporalMemory(
        connectedPermanence=0.50,
        minThreshold=1,
        seed=42
      )

      connections = tm.connections
      connections.createSegment(0)
      connections.createSynapse(0, 23, 0.6)
      connections.createSynapse(0, 37, 0.4)
      connections.createSynapse(0, 477, 0.9)

      connections.createSegment(0)
      connections.createSynapse(1, 49, 0.9)
      connections.createSynapse(1, 3, 0.8)

      connections.createSegment(1)
      connections.createSynapse(2, 733, 0.7)

      connections.createSegment(108)
      connections.createSynapse(3, 486, 0.9)

      activeCells = set([23, 37, 49, 733])

      self.assertEqual(tm.bestMatchingCell(tm.cellsForColumn(0),
                                           activeCells,
                                           connections),
                       (0, 0))

      self.assertEqual(tm.bestMatchingCell(tm.cellsForColumn(3),  # column containing cell 108
                                           activeCells,
                                           connections),
>                      (96, None))  # Random cell from column
E     AssertionError: Tuples differ: (103, None) != (96, None)
E
E     First differing element 0:
E     103
E     96
E
E     - (103, None)
E     ?  ^^^
E
E     + (96, None)
E     ?  ^^

tests/unit/nupic/research/temporal_memory_test.py:346: AssertionError
_____________________________________________________________ TemporalMemoryTest.testBurstColumns ______________________________________________________________
self = <tests.unit.nupic.research.temporal_memory_test.TemporalMemoryTest testMethod=testBurstColumns>

    def testBurstColumns(self):
      tm = TemporalMemory(
        cellsPerColumn=4,
        connectedPermanence=0.50,
        minThreshold=1,
        seed=42
      )

      connections = tm.connections
      connections.createSegment(0)
      connections.createSynapse(0, 23, 0.6)
      connections.createSynapse(0, 37, 0.4)
      connections.createSynapse(0, 477, 0.9)

      connections.createSegment(0)
      connections.createSynapse(1, 49, 0.9)
      connections.createSynapse(1, 3, 0.8)

      connections.createSegment(1)
      connections.createSynapse(2, 733, 0.7)

      connections.createSegment(108)
      connections.createSynapse(3, 486, 0.9)

      activeColumns = set([0, 1, 26])
      predictedColumns = set([26])
      prevActiveCells = set([23, 37, 49, 733])
      prevWinnerCells = set([23, 37, 49, 733])

      (activeCells,
       winnerCells,
       learningSegments) = tm.burstColumns(activeColumns,
                                           predictedColumns,
                                           prevActiveCells,
                                           prevWinnerCells,
                                           connections)

      self.assertEqual(activeCells, set([0, 1, 2, 3, 4, 5, 6, 7]))
>     self.assertEqual(winnerCells, set([0, 6]))  # 6 is randomly chosen cell
E     AssertionError: Items in the first set but not the second:
E     4
E     Items in the second set but not the first:
E     6

tests/unit/nupic/research/temporal_memory_test.py:192: AssertionError
__________________________________________________________ TemporalMemoryTest.testPickCellsToLearnOn ___________________________________________________________
self = <tests.unit.nupic.research.temporal_memory_test.TemporalMemoryTest testMethod=testPickCellsToLearnOn>

    def testPickCellsToLearnOn(self):
      tm = TemporalMemory(seed=42)

      connections = tm.connections
      connections.createSegment(0)

      winnerCells = set([4, 47, 58, 93])

      self.assertEqual(tm.pickCellsToLearnOn(2, 0, winnerCells, connections),
>                      set([4, 58]))  # randomly picked
E     AssertionError: Items in the first set but not the second:
E     93
E     Items in the second set but not the first:
E     58

tests/unit/nupic/research/temporal_memory_test.py:508: AssertionError
====================================================== 8 failed, 812 passed, 21 skipped in 173.53 seconds ======================================================

CC @rhyolight

subutai commented 9 years ago

I get almost exactly the same failures. Mine says 8 failed, 813 passed

I am using the latest from NuPIC (8b300a82) along with the associated test nupic.core commitish: 57a614d73e6

chetan51 commented 9 years ago

@breznak What version of nupic are you running?

rhyolight commented 9 years ago

I'm going to re-open until someone tests this.

jcasner commented 9 years ago

Thanks @rhyolight - this got closed by accident - these tests are definitely still failing

chetan51 commented 9 years ago

I think these failures are caused by an issue in nupic's Random, especially since its tests are failing (and Coordinate encoder uses it).

breznak commented 9 years ago

What version of nupic are you running?

Just retested on latest master of nupic and .core. And I could confirm the

tests/unit/nupic/math/nupic_random_test.py failing.

numenta / nupic-legacy

CoordinateEncoderTest.testEncodeRelativePositionsAndRadii test failure #2471

..