Closed SaiAshwin31 closed 6 years ago
Hi,
If it only is to be able to transfer numpy arrays between actors (and only among calvin-base runtimes) I would suggest an alternative approach. The default serialisation between runtimes is JSON but this is implemented with a plugin. We also support message pack, which is used towards calvin-constrained runtimes. So the suggestion is to introduce a new serialisation that supports numpy objects, you could build it on top of message pack and add new object types which message pack supports. Look in calvin/runtime/north/plugins/coders/messages. This has the advantage that when the actors passing the numpy object are on the same runtime no serialisation/deserialisation is made. This is on my wishlist.
To answer your original question. I'm not aware of that we actively prevent import so the error is likely due to the the serialize_sk is not in a recognised path. Anyway, to add e.g. such a module properly look at /calvin/runtime/north/calvinlib.py and /calvin/runtime/south/calvinlib for examples. Then change/add the calvin config to also include your new calvinlib, you have to list all that you need. You should use the develop branch not the master branch when making changes. The self.use has changed on develop branch, search for calvinlib in calvin/actorstore to see examples.
Thank you. I'll try the method that you have suggested. Also, you might want to update the "test-requirements.txt" file in the develop branch. I checked it out yesterday and tried installing the dependencies and a fresh installation of "MySQL-Python" complains about a missing config file.
Collecting MySQL-python (from -r test-requirements.txt (line 7))
Downloading MySQL-python-1.2.5.zip (108kB)
Complete output from command python setup.py egg_info:
sh: 1: mysql_config: not found
Traceback (most recent call last):
File "
installing the following resolved it.
sudo apt-get install libmysqlclient-dev
Hello, I was able to test out a custom serializer by adding one in: calvin/runtime/north/plugins/coders/messages, I however did this in the 'master' branch. I am currently unable to work with the dev branch because the moment I start two runtimes, I get a bunch of 'CRITICAL Twisted' errors as such:
2017-12-06 13:55:48,320 CRITICAL 5018-twisted: 'Did not received reply for msg id vBjpxiPpEY+gio3V8voa0uiVHVM= within 5 seconds'
2017-12-06 13:55:48,321 CRITICAL 5018-twisted: 'Did not received reply for msg id s6gkFe5bgGvNtA8JqpAjEDlmPj8= within 5 seconds'
2017-12-06 13:55:48,322 CRITICAL 5018-twisted: 'Did not received reply for msg id VzDXOEGcb6We0MaI7/gnet8PLvc= within 5 seconds'
2017-12-06 13:55:48,323 CRITICAL 5018-twisted: 'Did not received reply for msg id 06pLuS+LcRjByoYrblVoRsRD2aI= within 5 seconds'
2017-12-06 13:55:48,325 CRITICAL 5018-twisted: 'Did not received reply for msg id Xsq6Xu2Ohii95+TfchuGKYrlbvU= within 5 seconds'
2017-12-06 13:55:48,326 CRITICAL 5018-twisted: 'Did not received reply for msg id yH8cTnNtS3b29+YknU1E4Q4zCGY= within 5 seconds'
2017-12-06 13:55:48,328 CRITICAL 5018-twisted: 'Did not received reply for msg id bWVJrvKv8xJh37u3pjZHa39kHow= within 5 seconds'
2017-12-06 13:55:48,329 CRITICAL 5018-twisted: 'Did not received reply for msg id ijv+/RRxV1SPobd6s6DzfQitPBY= within 5 seconds'
2017-12-06 13:55:48,330 CRITICAL 5018-twisted: 'Did not received reply for msg id 5bDmJ5gJ0r5VcRetambprtp1Tjk= within 5 seconds'
2017-12-06 13:55:48,331 CRITICAL 5018-twisted: 'Did not received reply for msg id c7Cz+YkB26skE7c2wrlgqsr05Ak= within 5 seconds'
2017-12-06 13:55:48,332 CRITICAL 5018-twisted: 'Did not received reply for msg id VEWXB2soUasYFBN9RtJ+1nEtLjo= within 5 seconds'
I tried logging and tried to analyze what might be causing this, but did not seem to help. Is there any reason this might happen? any insights in resolving this would be appreciated.
That error is related to the DHT storage, which I don't think is different between develop and master. You get this error when there is connectivity issues between the DHT nodes, typically related to network issues, but the DHT is a bit of a problem. Are you building the serialiser on msgpack? If so then you should be aware that the DHT also use msgpack for the serialization, hence if you changed the module it would also affect the DHT. You could switch to local storage on one of the runtimes and then set the other runtime using it as proxy storage, see wiki for the configuration.
I'm closing this issue now. If you have further questions, feel free to open a new one.
Cheers.
Hello, I have a non-actor specific functionality that I use within an actor implementation, i.e. I am trying to use a custom serializer to serialize numpy arrays as JSON (basically a trained model). From the source code I understand that library specific functions can be instantiated within the setup() method in an actor implementation using the following:
I wanted to know if there was a "right" approach to adding these implementations into the calvin framework. Currently I am able to use it with a simple import statement and having the source code in the same folder as the actor, but I do get the following error:
2017-11-07 12:21:29,568 ERROR 4618-calvin.calvin.actorstore.store: Could not load python module Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/calvin-0.8-py2.7.egg/calvin/actorstore/store.py", line 150, in _load_pymodule pymodule = imp.load_source(name, path) File "/home/sai/PycharmProjects/calvin-iotx/applications/winturbine_predictive_maintenance/actors/wt/MVLRPredictor.py", line 4, in
import serialize_sk as sr
ImportError: No module named serialize_sk