search

numenta / nupic-legacy

Numenta Platform for Intelligent Computing is an implementation of Hierarchical Temporal Memory (HTM), a theory of intelligence based strictly on the neuroscience of the neocortex.
http://numenta.org/
GNU Affero General Public License v3.0
6.33k stars 1.56k forks source link

Serialize data in binary format, not ascii #3854

Closed ghost closed 5 years ago

ghost commented 5 years ago

I understand that these methods are depreciated, but they are much faster than capnproto....with this change its orders of magnitude faster.

Please see: https://stackoverflow.com/questions/23582489/python-pickle-protocol-choice

Edit: Looks like the contributor validator failed, I have already signed the paper work for this FYI, you got my fingerprints, iris scan etc

rhyolight commented 5 years ago

Trying to make sure the tests are running properly. Failures initially on both builds.

lscheinkman commented 5 years ago

@kyle-sorensen The integration tests are failing. Could you please run the integration tests and make sure they pass with the new protocol? Here is the command to run the tests:

 python setup.py test --pytest-args="--verbose --boxed unit integration"

It looks like you need to update the tests to open the pickled files in 'binary' mode

ghost commented 5 years ago

@lscheinkman Edit: I'm running those tests now, I'll let you know.

It looks like you need to update the tests to open the pickled files in 'binary' mode

https://docs.python.org/2/library/pickle.html#pickle.load

This function automatically determines whether the data stream was written in binary mode or not.

ghost commented 5 years ago

@lscheinkman Do I need a MySQL db running on localhost and 3306 to run those tests? Many of them failed on timeout errors...

OperationalError: (2003, "Can't connect to MySQL server on 'localhost' ([Errno 61] Connection refused)")

lscheinkman commented 5 years ago

@lscheinkman Do I need a MySQL db running on localhost and 3306 to run those tests? Many of them failed on timeout errors...

OperationalError: (2003, "Can't connect to MySQL server on 'localhost' ([Errno 61] Connection refused)")

Yes, the swarm integration tests require mysql server installed.

lscheinkman commented 5 years ago

@kyle-sorensen The specific test failing on windows is CheckpointStressTest::testCheckpoint with EOFError.

I believe it is caused by opening the saved file in text mode instead of binary mode. See https://github.com/numenta/nupic/blob/master/src/nupic/frameworks/opf/model.py#L390

ghost commented 5 years ago

@lscheinkman Thanks, tests now passing

lscheinkman commented 5 years ago

👍 LGTM