luispedro / jug

Parallel programming with Python
https://jug.readthedocs.io
MIT License
412 stars 62 forks source link

Cannot load empty strings #39

Closed mfeurer closed 8 years ago

mfeurer commented 8 years ago

Hi,

first of all, jug is a great module to distribute tasks on a cluster without too much overhead and it already helps me a lot.

While using it, I found that one cannot read results which contain empty strings in python 2.7.6 (not tested with python 3). Here is a minimal working example:

File jugtest.py

from jug import Task

def func():
    return ''
t = Task(func)

File jugtest_result.py

import jug
jug.init('jugtest.py', 'jugtest.jugdata')
import jugtest

print(jugtest.t.value())

Output:

$ jug execute jugtest.py
Executed      Loaded  Task name
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1           0 jugtest.func                                                                   
.............................................................................................................................................................................................................
1           0     Total 

$ python jugtest_result.py 
Traceback (most recent call last):
  File "jugtest_result.py", line 5, in <module>
    print(jugtest.t.value())
  File "/home/feurerm/virtualenvs/2016_dataset_metric_learning/local/lib/python2.7/site-packages/jug/task.py", line 113, in value
    return self.result
  File "/home/feurerm/virtualenvs/2016_dataset_metric_learning/local/lib/python2.7/site-packages/jug/task.py", line 108, in _get_result
    if not hasattr(self, '_result'): self.load()
  File "/home/feurerm/virtualenvs/2016_dataset_metric_learning/local/lib/python2.7/site-packages/jug/task.py", line 150, in load
    self._result = self.store.load(self.hash())
  File "/home/feurerm/virtualenvs/2016_dataset_metric_learning/local/lib/python2.7/site-packages/jug/backends/file_store.py", line 228, in load
    return decode_from(input)
  File "/home/feurerm/virtualenvs/2016_dataset_metric_learning/local/lib/python2.7/site-packages/jug/backends/encode.py", line 192, in decode_from
    return pickle.load(stream)
EOFError

This is not a big issue because I can just output a dummy string, but it would be great if this can be fixed or documented so other users don't have this issue again.

luispedro commented 8 years ago

Thanks for your report, I can confirm I see this behaviour.

The culprit is b4e2ba090fed86bccbb2a2f53d0f8cab5b3eeb5e. This commit introduced the use of HIGHEST_PROTOCOL for pickle()ing. Somehow, this results in an error. I will look into it more later