search

escattone / txpool

A persistent process pool in Python for use with Twisted.
MIT License
15 stars 3 forks source link

Project Development Status #1

Open jomido opened 8 years ago

jomido commented 8 years ago

Hi Ryan (@escattone),

Was playing around with this project today, and like it a lot. I'm wondering if you've any further plans for txpool, or have features/improvements in mind that haven't yet been implemented. If so, I am looking to work on something small-ish. Let me know!

Jonathan

escattone commented 8 years ago

Hi Jonathon!

Sorry for my delayed response, but the last week has been crazy busy! Thanks for your interest! It's something I wrote while working at Lucasfilm (in fact it's still in use at Lucasfilm as part of an internally-developed distributed work queue much like Celery), but I confess I haven't given it much thought or attention lately. If you have any ideas, let me know, and I'll think some more as well.

Cheers! Ryan

On Fri, Jul 22, 2016 at 8:39 AM, Jonathan Dobson notifications@github.com wrote:

Hi Ryan (@escattone https://github.com/escattone),

Was playing around with this project today, and like it a lot. I'm wondering if you've any further plans for txpool, or have features/improvements in mind that haven't yet been implemented. If so, I am looking to work on something small-ish. Let me know!

Jonathan

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/escattone/txpool/issues/1, or mute the thread https://github.com/notifications/unsubscribe-auth/ADkfza1iKQ0vPZyZ28T912uddcChHl3Yks5qYOQtgaJpZM4JS4PQ .

Ryan Johnson escattone@gmail.com http://www.linkedin.com/in/ryanjohnson5

jomido commented 8 years ago

Cool, thanks for the backgrounder! I did notice there is a limit to the size of data that can be passed back from a worker. The pipe gets closed, if I recall. I had the worker return a large string as a test, something like "0" * n, and hit a wall around n = 99993.

chrisbarber commented 7 years ago

For anyone wondering about this, txpool is using Twisted's NetstringReceiver for communicating pickled objects. NetstringReceiver defines MAX_LENGTH which defaults to 99999. Setting MAX_LENGTH to something larger in txpool's WorkerProtocol should allow transfer of arbitrarily large pickled objects, at your own risk.

chrisbarber commented 7 years ago

3