search

douban / dpark

Python clone of Spark, a MapReduce alike framework in Python
BSD 3-Clause "New" or "Revised" License
2.69k stars 534 forks source link

pymesos #40

Closed dangra closed 10 years ago

dangra commented 10 years ago

Hello, have you considered releasing dpark.pymesos as an standalone package?

I have used it with great success and now that mesos 0.20.0 is out it is possible to install mesos interfaces from mesos.interface to avoid copy-pasting more code.

I like pymesos doesn't require extra packages compared to https://github.com/wickman/pesos project.

thanks

windreamer commented 10 years ago

It's possible to make pymesos a standalone package.

However mesos.interface package seems to be little help for pymesos, as pymesos needs message_pb2.py imported, which contains internal interfaces and structs and is not released within mesos.interface package.

So I'm not sure whether releasing pymesos is beneficial. Any comment is welcome.

windreamer commented 10 years ago

And sorry for the late reply, it was Chinese Mid-Autumn Festival here. :)

dangra commented 10 years ago

Actually, I found what you said about messages_pb2.py while attempting to make pymesos an standalone package. Still, there are some benefits like having more people helping to keep it uptodate with mesos releases.

I worked on making an standalone pymesos package (for internal use but ready to be released), then I updated it for mesos 0.20.0 and implemented reconcileTasks() method to SchedulerDriver interface. I am very happy to contribute my work back to you but ideally in the form of a standalone package so we don't have to maintain two versions. thanks.

windreamer commented 10 years ago

OK, that's fine. I'm waiting for the releasing.

dangra commented 10 years ago

There you go https://github.com/dangra/pymesos.

dangra commented 10 years ago

I'm leaving for vacations tomorrow, feel free to move the repo to whatever place you think it is best. Also I think pymesos is a good name and it is available on pypi at the moment. thanks!

dangra commented 10 years ago

back, any chance you looked at the pymesos repo?

windreamer commented 10 years ago

looks great! I'v made a fork as douban/pymesos. maybe we should submit this package to pypi when it's ready. And I'm working on a new version of DPark to use this pymesos package instead of the hard-coded one, and making DPark compatible with Mesos 0.20. The new version of DPark still need more tests.

windreamer commented 10 years ago

We've made a repo https://github.com/douban/pymesos & uploaded the package to https://pypi.python.org/pypi/pymesos/0.0.1.

dangra commented 10 years ago

great!