search

numenta / nupic-legacy

Numenta Platform for Intelligent Computing is an implementation of Hierarchical Temporal Memory (HTM), a theory of intelligence based strictly on the neuroscience of the neocortex.
http://numenta.org/
GNU Affero General Public License v3.0
6.33k stars 1.55k forks source link

Evaluate options for a language-independent checkpoint/serialization format for the CLA #333

Closed rhyolight closed 10 years ago

rhyolight commented 11 years ago

The current CLA model checkpoint uses the pickle module. As we move towards multiple language support and more external model sharing, we should define a language-independent format for serializing the CLA.

The major objectives for this would be cross-language implementation (i.e. we don't have to create serialize functions separately for each language), speed, checkpoint size, and ease of development and versioning.

Status

iandanforth commented 11 years ago

Thrift is a PITA both conceptually and in implementation.

--Please excuse brevity, sent from phone.-- On Oct 24, 2013 5:01 PM, "Matthew Taylor" notifications@github.com wrote:

The current CLA model checkpoint uses the pickle module. As we move towards multiple language support and more external model sharing, we should define a language-independent format for serializing the CLA.

The major objectives for this would be cross-language implementation (i.e. we don't have to create serialize functions separately for each language), speed, checkpoint size, and ease of development and versioning.

Some options include:

  • Protocol buffers
  • Thrift
  • Cap'n proto - a little early in development but has some nice properties
  • Some form of binary JSON

— Reply to this email directly or view it on GitHubhttps://github.com/numenta/nupic/issues/333 .

scottpurdy commented 11 years ago

@iandanforth Yes I have heard (and experienced) similar opinions. Frank mentioned the same in JIRA I believe. Something like MessagePack is easy to use but is harder to maintain and doesn't work as well across languages (as far as I can tell).

I am leaning towards protobufs or similar. I quite like Cap'n Proto and there is active development but previously is was still a bit raw. One potential advantage of it over PB is that it has planned support for memory mapping that could make deserialize-run-serialize operations very fast (this is the current Grok use case).

scottpurdy commented 10 years ago

Here is my branch with initial Python spatial pooler test: https://github.com/scottpurdy/nupic/tree/sp-capnp

The files are in nupic/proto.

My initial results weren't very good. I suspect there is type conversion going on. It would probably be better to do the first tests in C++ where it is more obvious what is happening. And I wasn't doing thorough profiling, just saw that the time to create the model and the time to feed a record in were longer. From a theoretical standpoint, I am fairly confident that we can make serialization/deserialization and runtime both faster though.

utensil commented 10 years ago

:+1:

scottpurdy commented 10 years ago

There are currently four different options we want to measure times for: C++ and Python protocol buffers and C++ and Python Cap'n Proto buffers.

There are two ways to use these. In the simple scenario, we leave the code pretty much the same and simply create and populate the buffers when we need to serialize. The other is to actually use the buffer in memory during execution. This doesn't work for some fields like sparse matrices but it works for most everything else.

In the former case, we want to measure the times for the following operations:

In the case that we use the buffer in memory during execution, we would want to measure the time it takes to run records through in addition to the times for copying/serializaing/deserialization.

Finally, when doing these timing tests it is important to run some records through the pre-serialization and post-de-serialization objects and compare the results to ensure that everything is implemented correctly (wouldn't be fair timing tests is some pieces were left out!).

utensil commented 10 years ago

Yeah, seen the code and it's taking the approach of "actually use the buffer in memory during execution".

utensil commented 10 years ago

It would probably be better to do the first tests in C++ where it is more obvious what is happening. And I wasn't doing thorough profiling, just saw that the time to create the model and the time to feed a record in were longer.

It seems that Cap’n Proto does the encoding when feeding the data, and decoding on retrieval:

Cap’n Proto gets a perfect score because there is no encoding/decoding step. The Cap’n Proto encoding is appropriate both as a data interchange format and an in-memory representation, so once your structure is built, you can simply write the bytes straight out to disk!

So, I guess it's definitely not an option to use Cap’n Proto as internal structure of SP and such, it will slow things down.

vsinha commented 10 years ago

Captain Proto is currently C++11 only and I haven't been able to get it to link with the current nupic.core (or even Marek's C++11branch)

Performance with protocol buffers looks good: both the stages of creating the protocol buffer and serializing, as well as deserialization and loading variables back into the class fields were about 2 times faster than the current implementation. Right now my implementation doesn't use the protobuf object in memory throughout SP execution, but allocates and populates it when the save function is called.

h2suzuki commented 10 years ago

I'm wondering... what kind of API should we need for language bindings? Should it be an IDL kind of stuff to allow generic RPC --- procedural? or should it be a memcached or SQL kind of text/binary protocol to allow generic data access -- data oriented?

It also depends on the medium to communicate upon; dynamic linking mechanism or mere network packets. We can do binding in multiple levels. Does the efficiency matter? The speed can be optimized for throughput, latency, operations per second, ..., etc.

rhyolight commented 10 years ago

When I get back from vacation, I want to have a discussion about updating to C++11. That might change the implementation of this issue significantly.

utensil commented 10 years ago

There's #130 taking the approach of using google protocol buffer. Just wonder if anyone has noticed and evaluate FlatBuffers which is also the successor of protocol buffer and developed by google. At first glance, it also provides "Access to serialized data without parsing/unpacking" and has miscellaneous efficiency improvements just like Cap'n proto .

rhyolight commented 10 years ago

@scottpurdy Would you call this ticket complete?

utensil commented 10 years ago

rhyolight commented a day ago

@scottpurdy Would you call this ticket complete?

If it's complete, what's the conclusion?

scottpurdy commented 10 years ago

We don't have a decision on this yet. I think it makes sense to keep tracking here. I will create a follow up issue to track the implementation to be done after we finalize the decision.

scottpurdy commented 10 years ago

Now that we have a C++11 nupic.core I am going to attempt Capn Proto again.

scottpurdy commented 10 years ago

We have more motivation for this issue from #1231 which is a somewhat serious bug in NuPIC.

rhyolight commented 10 years ago

@scottpurdy Once C++11 is finished across nupic and nupic.core, the decision is to start a Cap'n Proto implementation, right?

scottpurdy commented 10 years ago

@rhyolight - that is my current plan, yes

rhyolight commented 10 years ago

Shall we close this yet?

Matt Taylor OS Community Flag-Bearer Numenta

On Fri, Oct 3, 2014 at 10:28 AM, Scott Purdy notifications@github.com wrote:

@rhyolight https://github.com/rhyolight - that is my current plan, yes

— Reply to this email directly or view it on GitHub https://github.com/numenta/nupic/issues/333#issuecomment-57827209.

rhyolight commented 10 years ago

Closing, assuming we're going with Cap'n Proto in #1336.