search

ipeterov / convenient-rpc

Convenient APIs for calling python functions remotely at many workers.
MIT License
0 stars 0 forks source link

Find a way to install multiple versions of one package to workers #10

Open ipeterov opened 8 years ago

ipeterov commented 8 years ago

Now workers use virtualenvs and virtualenv-api which don't allow them to be able to complete tasks that require different versions of one package.

If there is no readymade solution to that, we could probalby download modules from PyPi, unpack them in folder named after 'hash(package_name+version)', and import them by path. We could even write a python module for that. Probably isn't such a bad solution - workers will be in docker containers.

ipeterov commented 8 years ago

It maybe makes sense to marshal functions and pass whole functions instead of function desctiptions. But will this work with functions from packages (which may have C extensions)?

ipeterov commented 8 years ago

There is a cloud module that pickles functions into strings, which then can be unpickled and executed. Though, all the modules have to be installed AND imported. We could maybe make use of this module integrate it with the system that installs and imports packages from PyPi.

http://stackoverflow.com/questions/10048061/how-to-pickle-a-python-function-with-its-dependencies

ipeterov commented 8 years ago

Pickling funcions with cloudpickle implemented, but issue is still open - we have to implement the thing I originally suggested.