search

soravux / scoop

SCOOP (Scalable COncurrent Operations in Python)
https://github.com/soravux/scoop
GNU Lesser General Public License v3.0
636 stars 87 forks source link

next release? #60

Open joernhees opened 7 years ago

joernhees commented 7 years ago

scoop is a pretty cool lib for easy parallelization, which is why i use it in many of my projects...

However, there are many bugfixes in master that aren't in the latest release 0.7.1.1 and i often find myself "rediscovering" them as 0.7.1.1 is installed and not master...

Would it be possible to have a next release soonish?

soravux commented 7 years ago

The problem is that I don't have access to a cluster to perform my tests anymore and many features were added in by users for setups I don't have access to (e.g., SLURM). I believe there are a couple of problems in the current master branch of SCOOP that I sadly cannot test or fix. A lot of advanced features were added but not entirely tested. If the community in majority believes it is stable enough for a new PyPI release, I'll do it. Any testing report or pull request are gladly welcome.

maharjun commented 7 years ago

I am in the process of fixing scoop and rewriting certain sections in order to make the code and logic clearer. I will also probably include a markdown file detailing the logic of SCOOP as simply as possible. This way we can be sure of certain guarantees. I'm not familiar with unit test frameworks in Python but as I do have access to a cluster to test them I will gladly run them and confirm.

At the same time, at our institute we are considering the following upgrades to SCOOP:

  1. Dynamic resource allocation/deallocation via SLURM / otherwise (This can be done in the next version)

[sorry cant seem to think of more right now]

@joernhees If you can help me out with testing scoop on your cluster, then I think it may be possible to get something release worthy by the end of the next week.

micheles commented 7 years ago

We are considering SCOOP for running our computational engine (https://github.com/gem/oq-engine), however we are worried about the status of the project. Having a new release and proving that the project is maintained would help selling SCOOP. From the GitHub graphs this looks like an abandoned project, I see next to zero activity in the last three years.

maharjun commented 7 years ago

Currently, we are using scoop in TU Graz to run independent simulations of spiking neural networks. I have currently fixed many of scoop's bugs. Currently however, I'm afraid the software isn't release ready owing to the following:

  1. The cleanup of processes is not clean in the case of a keyboard interrupt or exception.
  2. There appears to be a slight issue where if the first host in the hostfile is not the host from which the simulations are run, the output is staggered
  3. I have not tested this code for a special case of using multiple brokers.

However, for the purpose of our simulations, none of the above are serious enough. Our current workaround for remaining processes is to check if parent exists and if not, self-destruct.

however, I would not recommend this software be made a dependency of production grade software.

As always, any help with the above bugs will be appreciated. Unfortunately, As I do not have write access to this repository, If you would like to contribute, you could do so to my repository as I am currently actively maintaining it to the best of my understanding.

https://github.com/IGITUGraz/scoop