EntilZha / PyFunctional

Python library for creating data pipelines with chain functional programming
http://pyfunctional.pedro.ai
MIT License
2.41k stars 132 forks source link

Async Support #186

Closed maestro-1 closed 1 year ago

maestro-1 commented 1 year ago

PyFunctional is an excellent library for dealing with things in a chained manner which is really nice for keeping track and kind of working in a functional way. I think everyone should use it. However, there doesn't seem to be a way to work with asynchronous Python code. Is this outside the mission of Pyfunctional or is that just happenstance from no one being available to implement said functionality?

EntilZha commented 1 year ago

Hi @maestro-1, can't speak for others, but I haven't had much time to add new features in a while (TBF, the library is fairly mature).

That said, async is most helpful for code where you are not CPU bound and instead spend a lot of time waiting for specific events. E.G., waiting for an HTTP request to return. At least in my experience, most of what I'd use PyFunctional for doesn't really match this case and is CPU bound if anything (one main reason I reach for PyFunctional is an easy way to parallelize).

Is there a particular use-case I might be missing where some form of async support would be helpful?

maestro-1 commented 1 year ago

Hello @EntilZha Well, I can see support for reading files and connecting to databases which is very much IO bound. Admittedly the support for reading a file with async is a bit messy and not natively supported(and including that may here may end up being a pain to maintain), but that's not the same for databases.

Now, there will very likely never be a need to include support for postgres and the likes cause sqlalchemy exists and accomplishes this for the most part, but for SQLite database interactions(which is supported by the PyFunctional) this would go a long way. This can also be extended into other libraries like Pg and Mysql if they ever get included here.

EntilZha commented 1 year ago

@maestro-1 As usual, I am open to contributions. As with other larger feature additions though, I'd suggest first speccing out what it will look like and what code would need to be change, before putting in the work to do it.

My suspicion is that first challenge is that some data sources might not have easy async support.

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.