search

rheem-ecosystem / rheem-api-python

0 stars 0 forks source link

Does Python platform need op. like 'filter' or 'map' ? #1

Closed NKcqx closed 4 years ago

NKcqx commented 4 years ago

Hi~ I notice that you're adding python platform to rheem, is that right ? and I also notice that you've added some operators like 'filter', 'map' but I'm not sure whether these kind op. are feaquently used in python scenario, since neither TF nor pytorch have those ( or maybe they have but not commonly used ? ) Bacause in my naive opnion, we usually use python to do some math compute, so how do you think of that ?

ro-pardo commented 4 years ago

Hi, thank you very much for your feedback.

Yes, we are adding Python to Rheem. Our first goal is to enable the users to create their Rheem pipelines with Python. At this stage of the development, the team expect to use the typical database operations, to allow Python users to process their data in parallel. Keep in mind that inside the UDF of this operators (like filter or map), users will be able to use python libraries functions.

Rheem has been made to solve different kind of Use cases, having Big Data Flows; so, the project is not specific for the ML field. Therefore, we are welling to help also other kind of Rheem users. However, we recognize that as you said, in Python the Data Science community using these technologies is one of the most relevant ones, and we must provide specific tools for them.

Anyways, if you have in mind a better approach to proceed with this integration or recommendations to help us, we are happy to get in contact with you. We strongly recommend you to join us in the Rheem Community on Slack.

rheem-community.slack.com

NKcqx commented 4 years ago

OK I see, this python package is for the one who's still going to build Big Data Flows but hacking with python. Thank you for the patient explanation. And also thank you for the invitation, always wanna find a way to keep contact with you. But it seems there's some trouble logging in: image

NKcqx commented 4 years ago

So sorry to bother again, I still can't join in Slack. Can u plz send a invitation or something? My email address is nkucqx@gmail.com

berttty commented 4 years ago

So sorry to bother again, I still can't join in Slack. Can u plz send a invitation or something? My email address is nkucqx@gmail.com

I just send you an invitation, but thanks to inform about the issue of the sign in I will see what is producing the issue :D

Just in case you can send an email to oss@rheem.io

NKcqx commented 4 years ago

It works THX~! So happy to join the community !