search

soravux / scoop

SCOOP (Scalable COncurrent Operations in Python)
https://github.com/soravux/scoop
GNU Lesser General Public License v3.0
635 stars 87 forks source link

SCOOP crashes depending on network connection. #45

Open erp12 opened 8 years ago

erp12 commented 8 years ago

I have been using SCOOP with DEAP for while now, and I really enjoy how simple it is. I have run into one annoying problem that I am hoping is a simple fix.

When I run a script that uses SCOOP at home (or most other internet connections) it works fine. I was running into trouble when I tried to run the same scripts on my work's internet connection. When I switched to the public internet of the building next door, it began working again. I get the same SCOOP error when I am on my work's internet when I disable my internet connection all together.

I believe I read somewhere that SCOOP uses networking to communicate between workers. I also know that SCOOP has the capability of pinging out to other machines on the local network to see if it is being called as part of a cluster. I assume something in this realm is the source of the issue.

I am hoping there is a simple way to tell SCOOP to only use the cores on the local machine, and not to attempt to reach out across the network, as some connections (or no connections) make that impossible.

If such a workaround does or does not exist, I think it would be nice for the error reported to the user to explain more about what has gone wrong, and perhaps point to specific conditions which SCOOP must be run under.

Or perhaps I am completely wrong about what is happening here. Any help would be much appreciated.

Below is the error that I get when I run a script that uses SCOOP (python -m scoop my_file.py) at work or with my internet connection disabled:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scoop/__main__.py", line 18, in <module>
    from .launcher import main
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scoop/launcher.py", line 31, in <module>
    from scoop import utils
soravux commented 8 years ago

Thanks for reaching out. I'm happy to see that SCOOP works for you at home. I have a guess about what's happening when you disable your internet connection: SCOOP queries the networking module to get an identifier of the local system. When you disable every interface, this particular query fails (as it cannot get an identifier). By the way, I'm not sure I understand what you mean by your work's internet connection which is a disabled interface?

As to why it isn't reported more appropriately in a Python traceback, I am not sure. My best bet would be that most of SCOOP runs into a 'sandbox' generated by runpy. When the identity query call fails, if it does not fail at runtime but import time, that would explain what you are seeing (and runpy eats away the error message).

Now, as per the fix, I am of course welcoming any pull requests. These times, I am a bit overwhelmed with other projects, so I can't guarantee a quick fix on my part. If you are ready to mess a bit with the code, here's what I would try: on scoop/utils.py, try replacing the call to socket.getfqdn() to a static string of your choice. I believe it could be the culprit in your problem.