build error on linux32 bit - missing apr_base64.h

[jkthompson]

Hi, just forked nupic and looking forward to trying it out. Building (via build.sh) on 32-bit Linux (Ubuntu 12.04) I get the following error:

make: *** [all-recursive] Error 1
/home/jthompson/Projects/Numenta/nupic/nta/utils/StringUtils.cpp:30:30: fatal error: apr-1/apr_base64.h: No such file or directory
compilation terminated.

I can can get a successful build by copying the following files from external/linux64/include/apr-1/ to external/linux32/include/apr-1/

1. apr_base64.h
2. apu.h

Not sure if this is the best solution.

Testing:

htmtest - all pass testeverything - all pass run_tests.sh - failed tests/unit/py2/nupic/encoders/nonuniformscalar_test.py:38: NonUniformScalarEncoderTest.testNonUniformScalarEncoder FAILED

Not sure if the failed test is related to the build issue or not.

Let me know if you'd like me to do any further testing. I'm happy to help.

[breznak]

Hi,

thanks for the report, we didnt do much testing on linux32 -bit platform, so you're touching unexplored grass there. Please make a PR with the two missing files that fix it.

I would thing the EncoderTest is unrelated to the build. What details does the err give?

Cheers, Mark

On Thu, Nov 28, 2013 at 9:43 AM, jkthompson notifications@github.comwrote:

Hi, just forked nupic and looking forward to trying it out. Building (via build.sh) on 32-bit Linux (Ubuntu 12.04) I get the following error:

make: *\ [all-recursive] Error 1 /home/jthompson/Projects/Numenta/nupic/nta/utils/StringUtils.cpp:30:30: fatal error: apr-1/apr_base64.h: No such file or directory compilation terminated.

I can can get a successful build by copying the following files from 'external/linux64/include/apr-1/' to 'external/linux32/include/apr-1/' 1) apr_base64.h 2) apu.h

Not sure if this is the best solution.

Testing: htmtest - all pass testeverything - all pass run_tests.sh - failed tests/unit/py2/nupic/encoders/nonuniformscalar_test.py:38: NonUniformScalarEncoderTest.testNonUniformScalarEncoder FAILED

Not sure if the failed test is related to the build issue or not.

Let me know if you'd like me to do any further testing. I'm happy to help.

— Reply to this email directly or view it on GitHubhttps://github.com/numenta/nupic/issues/445 .

Marek Otahal :o)

[jkthompson]

Hi Mark,

Just submitted a PR (#446) for this. Now sure how to link the PR to this issue, but hopefully it's easy to find. Agreed about the EncoderTest. I dumped the output of the test to a file, but have not investigated much yet. What would be the best way to share the file?

-Jeff

[jta1954]

I am in the same situation as jkthompson. I have a 1 GB computer with ubuntu 12.04. When I build, I got the same problem and solved it with the same solution. I also got the same error when running run_tests.py. I looked further, to see if I could solve the problem. This is what I got:

The program stops in the StressTest of tests/unit/py2/numpic/encoders/nonuniformscalar_test.py There it calls NonUniformScalarEncoder in lib/python2.7/site-packages/nupic/encoders/nonuniformscalar.py There is an infinite loop:

while True:
  # Use the inverse cumulative probability mass function
  # to compute the bin endpoint
  bins[binIndex, 0] = startValue
  cumBinWeight += avgBinWeight
  endValue = numpy.interp(cumBinWeight, xp=cumWeights, fp=sortedValues)
  bins[binIndex,1] = endValue

  if verbosity > 1:
      print "Start Value:%.2f EndValue:%.2f" %(startValue, endValue)

  if abs(cumWeights[-1] - cumBinWeight) < 1e-10:
    break

  startValue = endValue
  binIndex += 1

It stays in the loop with the following constant values: startValue and endValue are 5.00 cumWeights[-1] is 85, cumBinWeight is 0 And when binIndex = 94 is the progam stops with an IndexError

The problem is that the tests stop after one fail. After commenting out the problem, I got new fails, in lib/python2.7/site-packages/nupic/support/unittesthelpers/algorithm_test_helpers.py:46. Maybe because the seed is too big for a 32bits computer.

Is there a way to keep the tests running, after a failed test? I tried putting the exitStatus in run_tests.py always on 0, but that didn't work.

greetings: Jos Theelen

[jta1954]

I did some further testing. In total I get 3 errors. The first is described in the previous post. The other two are in:

• tests/unit/py2/nupic/research/flat_spatial_pooler_compatability_test.py
• tests/unit/py2/nupic/research/spatial_pooler_compatability_test.py

Both errors look the same. Both use the function getNumpyRandomGenerator(seed) in lib/python2.7/site-packages/nupic/support/unittesthelpers/algorithm_test_helpers.py and crash with ValueError: object of too small depth for desired array.

Maybe the seed is too big for a 32bit computer. Because I see numbers like 13858082487L and 13859005505L.

greetings: Jos Theelen

[breznak]

Hi Jeff,

thanks for the PR! it looks good, I'm glad we finally have someone to properly use and test the 32bit code.

Jeff, Jos,

thanks for your debugging guys! You could try to put a "return 42" at the first line of the offending function, to confirm the problem dwells there. I've found something similar mentioned at http://stackoverflow.com/questions/9534552/seeding-and-reusing-python-random-seeds so maybe we could find an inspiration for a fix.

Cheers, Mark

On Sun, Dec 1, 2013 at 1:47 PM, jta1954 notifications@github.com wrote:

I did some further testing. In total I get 3 errors. The first is described in the previous post. The other two are in: tests/unit/py2/nupic/research/flat_spatial_pooler_compatability_test.py tests/unit/py2/nupic/research/spatial_pooler_compatability_test.py

Both erros look the same. Both use the function getNumpyRandomGenerator(seed) in

lib/python2.7/site-packages/nupic/support/unittesthelpers/algorithm_test_helpers.py and crash with "ValueError: object of too small depth for desired array".

Maybe the seed is too big for a 32bit computer. Because I see numbers like 13858082487L and 13859005505L.

greetings: Jos Theelen

— Reply to this email directly or view it on GitHubhttps://github.com/numenta/nupic/issues/445#issuecomment-29573065 .

Marek Otahal :o)

[jta1954]

What I did was removing the extension (".py") of the failing functions, so they weren't be tested. And when I did that with those 3 functions, the run_tests.py run all to the end without failings. I will look further into the sources.

greetings: Jos Theelen

[jta1954]

When I adapt the function getNumpyRandomGenerator(seed) (in lib/python2.7/site-packages/nupic/support/unittesthelpers/algorithm_test_helpers.py) by adding an extra line "seed = 421", the errors in tests/unit/py2/nupic/research/flat_spatial_pooler_compatability_test.py and tests/unit/py2/nupic/research/spatial_pooler_compatability_test.py disappear. So I think the seed-values put in this function are too big.

def getNumpyRandomGenerator(seed):
  """
  Return a numpy random number generator with the given seed.
  If seed is None, set it randomly based on time. Regardless we log
  the actual seed and stack trace so that test failures are replicable.
  """
  if seed is None:
    seed = int(time.time()*10)
  seed = 421                                            <===== line added
  print "Seed set to:",seed,"called by",
  callStack = traceback.extract_stack(limit=3)
  print callStack[0][2],"line",callStack[0][1],"->",callStack[1][2]
  return numpy.random.RandomState(seed)

[jkthompson]

Thanks Mark, happy to help.

Jos, nice debugging work! You're way ahead of me on this. I'm still trying to come up to speed on Python (I'm more of a C++ guy). If you have any fixes you want me to test just let me know, and I'll do the same.

-Jeff

[subutai]

Nice debugging! Can you try replacing that line with:

seed = int((time.time()%10000)*10)

and see if that fixes it? If so, we can make that change.

Thanks!

[jta1954]

I wasn't sure which line I had to replace, the line after "if seed is None:" or the line I added myself. I tried both, and both worked. So you can change the line after "if seed is None:". If I understand it correctly, it also means that the function only failed, when the seed was put to None.

greetings: Jos Theelen

PS, I will also look at the other error(s), but I first want to see if I can get hotgym.py to work.

[breznak]

Hi Jos,

the line "seed = 421" was to test the error really is in the seed value being too big, which your testing proved. So you should re-remove that line and try if the one right after "if .. None" as Subutai suggested. If it works for you, please submit a PR and this could be closed successfully.

Thanks a lot, Mark

Marek Otahal :o)

[jta1954]

I don't know how (yet) to submit a PR, but maybe subutai could do it, because he came with the solution.

greetings: Jos Theelen

[rhyolight]

@jta1954 You can create a pull request against Jeff's original pull request by forking Jeff's fork, then creating a branch from his apr_base64 branch. Once you've pushed your local changes into a remote branch on github, you'll have the option to create a new PR against Jeff's apr_base64 branch, which he can then review and accept.

[jkthompson]

Hi Jos,

I'm a beginner at pull requests myself, I found this link helpful.

Also, did you happen to find a fix for the first bug you identified in nonuniformscalar.py?

This one seems to be unrelated to the random seed issue. I was going to look into this, unless it is already solved.

[subutai]

@jta1954 Hi Jos, I can go ahead and push a fix for this.

[jkthompson]

Hi, I found a fix for the first test failure that Jos identified in the StressTest of tests/unit/py2/numpic/encoders/nonuniformscalar_test.py. The problem was that avgBinWeight was being assigned to 0 on line 84 of nonuniformscalar.py, due to integer division and nBins being larger than cumWeights[-1].

avgBinWeight = cumWeights[-1] / nBins #old line

I changed the line to perform floating point division:

avgBinWeight = cumWeights[-1] / float(nBins) # new line

An alternative to this float cast is calling 'from future import division' at the top of the file, which changes the behavior of '/' to do floating point division by default. Apparently floating point division is the default behavior in Python 3, but not Python 2, http://stackoverflow.com/questions/1267869/how-can-i-force-division-to-be-floating-point-in-python

After this change, I was able to run all three test suites with zero test failures. I'm not sure why, but I did not observe the additional errors in flat_spatial_pooler_compatability_test.py and spatial_pooler_compatability_test.py that Jos encountered.

Any objections if I push this change to my existing PR (#446)?

[jta1954]

Nice to see you solved that first problem. I will test it at home.

The other problems happened when the seed was None. Then it takes some value depending on the internal time of the computer. In my computer those were numers like 13859005505. I assume that the internal time of your computer was a lot lower, so you seed didn't became too big. That's my guess on the situation.

Thanks subutai for doing the PR for me.

greetings: Jos Theelen

[jta1954]

Yes, I tested it at home and your change worked for me, so that solved the first problem of run_tests.py. It now runs without any problems. Brilliant.

greetings: Jos Theelen

[subutai]

@jkthompson Nice debugging! That sounds like a good fix, so feel free to add that to your PR. I already pushed the fix to getNumpyRandomGenerator in a parallel PR (#453) I'm working on so don't include that. It hasn't been merged in yet, but should be soon.

[jkthompson]

Thanks @subutai I just pushed the fix. Thanks to Jos for the confirmation.

[subutai]

@jkthompson @jta1954 FYI, my fix to getNumpyRandomGenerator just got merged into master so we're good to go there.

[jkthompson]

@subutai Thanks for the update. Do I need to do anything else to my PR (#446) to get it merged in?

[subutai]

@jkthompson No, I think that's it! Looks like Scott approved it, so I just merged it.

[rhyolight]

:thumbsup: Thanks guys!