search

numenta / nupic-legacy

Numenta Platform for Intelligent Computing is an implementation of Hierarchical Temporal Memory (HTM), a theory of intelligence based strictly on the neuroscience of the neocortex.
http://numenta.org/
GNU Affero General Public License v3.0
6.34k stars 1.55k forks source link

Serialization failed: Exceeded message traversal limit. See capnp::ReaderOptions. #3808

Closed ghost closed 6 years ago

ghost commented 6 years ago 
Traceback (most recent call last):
  File "/opt/project/sample_processor.py", line 203, in <module>
    SampleProcessorWorker().run()
  File "/opt/project/sample_processor.py", line 71, in run
    anomaly_model = self.repo.checkout(self.model_id)
  File "/opt/project/modelrepo/cassandra_repo.py", line 82, in checkout
    htm_model, anomaly_likelihood_helper = self.unpackage_models(model.htm_buffer, model.al_buffer)
  File "/opt/project/modelrepo/model_repo.py", line 44, in unpackage_models
    htm_model = AnomalyModel.deserialize_htm(zlib.decompress(packaged_htm_buffer))
  File "/opt/project/anomalymodel.py", line 45, in deserialize_htm
    return HTMPredictionModel.read(reader)
  File "/usr/local/lib/python2.7/site-packages/nupic/frameworks/opf/htm_prediction_model.py", line 1339, in read
    network = Network.read(proto.network)
  File "/usr/local/lib/python2.7/site-packages/nupic/bindings/engine_internal.py", line 1286, in read
    instance.convertedRead(proto)
  File "/usr/local/lib/python2.7/site-packages/nupic/bindings/engine_internal.py", line 1306, in convertedRead
    self._initFromCapnpPyBytes(proto.as_builder().to_bytes()) # copy * 2
  File "/usr/local/lib/python2.7/site-packages/nupic/bindings/engine_internal.py", line 1314, in _initFromCapnpPyBytes
    return _engine_internal.Network__initFromCapnpPyBytes(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/nupic/bindings/engine_internal.py", line 2993, in readPyRegion
    return getattr(regionCls, methodName)(pyRegionProto)
  File "/usr/local/lib/python2.7/site-packages/nupic/bindings/regions/PyRegion.py", line 360, in read
    return cls.readFromProto(regionImpl)
  File "/usr/local/lib/python2.7/site-packages/nupic/regions/sdr_classifier_region.py", line 358, in readFromProto
    instance._sdrClassifier = SDRClassifierFactory.read(proto)
  File "/usr/local/lib/python2.7/site-packages/nupic/algorithms/sdr_classifier_factory.py", line 68, in read
    return FastSDRClassifier.read(proto.sdrClassifier)
  File "/usr/local/lib/python2.7/site-packages/nupic/bindings/algorithms.py", line 3112, in read
    instance.convertedRead(proto)
  File "/usr/local/lib/python2.7/site-packages/nupic/bindings/algorithms.py", line 3138, in convertedRead
    self._initFromCapnpPyBytes(proto.as_builder().to_bytes()) # copy * 2
  File "/usr/local/lib/python2.7/site-packages/nupic/bindings/algorithms.py", line 3158, in _initFromCapnpPyBytes
    return _algorithms.SDRClassifier__initFromCapnpPyBytes(self, *args, **kwargs)
RuntimeError: /root/bamboo-agent-home/xml-data/build-dir/NUP-CORE-NCRM/build/scripts/ThirdParty/Source/CapnProto/src/capnp/arena.c++:106: failed: Exceeded message traversal limit.  See capnp::ReaderOptions.
stack: 0x7ff91a470cfe 0x7ff91a450121 0x7ff91a45f37a 0x7ff91a464ff3 0x7ff91a44c74f 0x7ff91a44c7ed 0x7ff91a457fb1 0x7ff91a45d237 0x7ff91a45cb90 0x7ff91a454301 0x7ff91a363a7a 0x7ff91a3bf62a 0x7ff92cce8b69 0x7ff92ccecb80 0x7ff92cce9b88 0x7ff92cce9c3e
ghost commented 6 years ago

I think that in this case our serialized model might have been corrupted after saving it. I'm not sure this is an issue with the serialization or not. We see it sporadically and can't duplicate it.