soravux / scoop

SCOOP (Scalable COncurrent Operations in Python)
https://github.com/soravux/scoop
GNU Lesser General Public License v3.0
625 stars 88 forks source link

TypeError: cannot serialize 'greenlet.greenlet' object #57

Open mokawi opened 7 years ago

mokawi commented 7 years ago

For some reason, I'm getting an error because python is trying to serialize greenlets:

Traceback (most recent call last):
  File "/home/apps/Logiciels/Python/python-3.5.1/lib/python3.5/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/apps/Logiciels/Python/python-3.5.1/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/lochar/python/lib/python3.5/site-packages/scoop/bootstrap/__main__.py", line 302, in <module>
    b.main()
  File "/home/lochar/python/lib/python3.5/site-packages/scoop/bootstrap/__main__.py", line 92, in main
    self.run()
  File "/home/lochar/python/lib/python3.5/site-packages/scoop/bootstrap/__main__.py", line 290, in run
    futures_startup()
  File "/home/lochar/python/lib/python3.5/site-packages/scoop/bootstrap/__main__.py", line 271, in futures_startup
    run_name="__main__"
  File "/home/lochar/python/lib/python3.5/site-packages/scoop/futures.py", line 64, in _startup
    result = _controller.switch(rootFuture, *args, **kargs)
  File "/home/lochar/python/lib/python3.5/site-packages/scoop/_control.py", line 230, in runController
    execQueue.sendResult(future)
  File "/home/lochar/python/lib/python3.5/site-packages/scoop/_types.py", line 384, in sendResult
    self.socket.sendResult(future)
  File "/home/lochar/python/lib/python3.5/site-packages/scoop/_comm/scoopzmq.py", line 319, in sendResult
    pickle.HIGHEST_PROTOCOL,
TypeError: cannot serialize 'greenlet.greenlet' object

I'm not using greenlets, so I'm not sure why I'm getting this. I'm really trying to get started with scoop—I've never used mapreduce nor done parallelized code before, so I might also be making an obvious error. Here is my code:

import os
import re

from scoop import futures as fut

def get_sideeffect_sentences(f):
    sideeffect = re.compile(r"\bside\s+effects?\b")
    tagsplit =  re.compile(r"\s*<[hp]>\s*")
    for ln in open(f):
        for sent in tagsplit.split(ln):
            if sideeffect.search(sent.lower()):
                yield sent

def writeout(x, y):
    print(y, file=outf)

if __name__ == "__main__":
    scratch = os.getenv("SCRATCH")
    outf = open(scratch + "/now/sideeffect_sents.txt", "w")

    lsfiles = [ scratch + "/now/text/" + f for f in os.listdir(scratch + "/now/text")  ]
    fut.mapReduce(get_sideeffect_sentences, writeout, lsfiles)

    outf.close()

The command is "python -m scoop -n 8". I noticed there are Québécois-sounding last names in the docs, so perhaps it might help to say I've been trying that on Briarée (Calcul Québec computer).

yolsever commented 1 year ago

I am getting the same error